Generic SCSI Target Middle Level for Linux Vladislav Bolkhovitin Version 0.9.5 2006/12/01, actual for SCST 0.9.5 and later This document describes SCSI target mid-level for Linux (SCST), its architecture and drivers from the driver writer's point of view. Introduction

SCST is a SCSI target mid-level subsystem for Linux. It is designed to provide unified, consistent interface between SCSI target drivers and Linux kernel and simplify target drivers development as much as possible. It has the following features: Very low overhead, fine-grained locks and simplest commands processing path, which allow to reach maximum possible performance and scalability that close to theoretical limit. Incoming requests can be processed in the caller's context or in one of the internal SCST's tasklets, therefore no extra context switches required. Complete SMP support. Undertakes most problems, related to execution contexts, thus practically eliminating one of the most complicated problem in the kernel drivers development. For example, target drivers for Marvell SAS adapters or for InfiniBand SRP are less 3000 lines of code long. Performs all required pre- and post- processing of incoming requests and all necessary error recovery functionality. Emulates necessary functionality of SCSI host adapter, because from a remote initiator's point of view SCST acts as a SCSI host with its own devices. Some of the emulated functions are the following: Generation of necessary UNIT ATTENTIONs, their storage and delivery to all connected remote initiators (sessions). RESERVE/RELEASE functionality. CA/ACA conditions. All types of RESETs and other task management functions. REPORT LUNS command as well as SCSI address space management in order to have consistent address space on all remote initiators, since local SCSI devices could not know about each other to report via REPORT LUNS command. Additionally, SCST responds with error on all commands to non-existing devices and provides access control (not implemented yet), so different remote initiators could see different set of devices. Other necessary functionality (task attributes, etc.) as specified in SAM-2, SPC-2, SAM-3, SPC-3 and other SCSI standards. Device handlers architecture provides extra reliability and security via verifying all incoming requests and allows to make any additional requests processing, which is completely independent from target drivers, for example, data caching or device dependent exceptional conditions treatment. Interoperability between SCST and local SCSI initiators (like sd, st) is the additional issue that SCST is going to address (it is not implemented yet). It is necessary, because local SCSI initiators can change the state of the device, for example RESERVE the device, or some of its parameters and that would be done behind SCST, which could lead to various problems. Thus, RESERVE/RELEASE commands, locally generated UNIT ATTENTIONs, etc. should be intercepted and processed as if local SCSI initiators act as remote SCSI initiators connected to SCST. This feature requires some the kernel modification. Since in the current version it is not implemented, SCST and the target drivers are able to work with any unpatched 2.4 kernel version. Interface between SCST and the target drivers is based on work, done by University of New Hampshire Interoperability Labs (UNH IOL). All described below data structures and function could be found in . Terms and Definitions

SCST Architecture

SCST accepts commands and passes them to SCSI mid-level at the same way as SCSI high-level drivers (sg, sd, st) do. Figure 1 shows interaction between SCST, its drivers and Linux SCSI subsystem.

Interaction between SCST, its drivers and Linux SCSI subsystem.
Target driver registration

To work with SCST a target driver must register its template in SCST by calling scst_register_target_template(). The template lets SCST know the target driver's entry points. It is defined as the following: Structure scst_tgt_template

struct scst_tgt_template { int sg_tablesize; const char name[15]; unsigned unchecked_isa_dma:1; unsigned use_clustering:1; unsigned xmit_response_atomic:1; unsigned rdy_to_xfer_atomic:1; unsigned report_aen_atomic:1; int (* detect) (struct scst_tgt_template *tgt_template); int (* release)(struct scst_tgt *tgt); int (* xmit_response)(struct scst_cmd *cmd); int (* rdy_to_xfer)(struct scst_cmd *cmd); void (*on_free_cmd) (struct scst_cmd *cmd); void (* task_mgmt_fn_done)(struct scst_mgmt_cmd *mgmt_cmd); void (* report_aen)(int mgmt_fn, const uint8_t *lun, int lun_len); int (*proc_info) (char *buffer, char **start, off_t offset, int length, int *eof, struct scst_tgt *tgt, int inout); } Where: = 0 to signify the number of detected target adapters. A negative value should be returned whenever there is an error. Must be defined. If the driver needs to create additional files in its /proc subdirectory, it can use Functions More about As already written above, function If Target driver registration functions scst_register_target_template()

Function int scst_register_target_template( struct scst_tgt_template *vtt) Where: Returns 0 on success or appropriate error code otherwise. scst_register()

Function struct scst_tgt *scst_register( struct scst_tgt_template *vtt) Where: Returns target structure based on template vtt or NULL in case of error. Target driver unregistration

In order to unregister itself target driver should at first call scst_unregister()

Function void scst_unregister( struct scst_tgt *tgt) Where: scst_unregister_target_template()

Function void scst_unregister_target_template( struct scst_tgt_template *vtt) Where: SCST session registration

When target driver determines that it needs to create new SCST session (for example, by receiving new TCP connection), it should call struct scst_session *scst_register_session( struct scst_tgt *tgt, int atomic, const char *initiator_name, void *data, void (*result_fn) ( struct scst_session *sess, void *data, int result)); Where: A session creation and initialization is a complex task, which requires sleeping state, so it can't be fully done in interrupt context. Therefore the "bottom half" of it, if Session registration when

Session registration when
SCST session unregistration

SCST session unregistration basically is the same, except that instead of atomic parameter there is void scst_unregister_session( struct scst_session *sess, int wait, void (* unreg_done_fn)( struct scst_session *sess)) Where: All outstanding commands will be finished regularly. After The commands processing and interaction between SCST and its drivers

The commands processing by SCST started when target driver calls If the command required no data transfer, it will be passed to SCSI mid-level directly or via device handler's If the command is a If the command is a When the command is finished by SCSI mid-level, device handler's The commands processing flow Additionally, before calling Expected transfer length and direction via The commands processing functions scst_rx_cmd()

Function struct scst_cmd *scst_rx_cmd( struct scst_session *sess, const uint8_t *lun, int lun_len, const uint8_t *cdb, int cdb_len, int atomic) Where: scst_cmd_init_done()

Function void scst_cmd_init_done( struct scst_cmd *cmd, int pref_context) Where: scst_rx_data()

Function void scst_rx_data( struct scst_cmd *cmd, int status, int pref_context) Where: Parameter scst_tgt_cmd_done()

Function void scst_tgt_cmd_done( struct scst_cmd *cmd) Where: The commands processing context

Execution context often is a major problem in the kernel drivers development, because many contexts, like IRQ one, greatly limit available functionality, therefore require additional complex code in order to pass processing to more simple context. SCST does its best to undertake most of the context handling. On the initialization time SCST creates for internal command processing as many threads as there are processors in the system or specified by user via Directly, i.e. in the caller's context, without limitations Directly atomically, i.e. with sleeping forbidden In the SCST's internal per processor or per session thread In the SCST's per processor tasklet The target driver sets this context as pref_context parameter for Preferred context constants

There are the following preferred context constants: Task management functions

There are the following task management functions supported: scst_rx_mgmt_fn_tag()

Function int scst_rx_mgmt_fn_tag( struct scst_session *sess, int fn, uint32_t tag, int atomic, void *tgt_specific) Where: Returns 0 if the command was successfully created and scheduled for execution, error code otherwise. On success, the completion status of the command will be reported asynchronously via scst_rx_mgmt_fn_lun()

Function int scst_rx_mgmt_fn_lun( struct scst_session *sess, int fn, const uint8_t *lun, int lun_len, int atomic, void *tgt_specific); Where: Returns 0 if the command was successfully created and scheduled for execution, error code otherwise. On success, the completion status of the command will be reported asynchronously via Device specific drivers (device handlers)

Device specific drivers are plugins for SCST, which help SCST to analyze incoming requests and determine parameters, specific to various types of devices. Device handlers are intended for the following: To get data transfer length and direction directly from CDB and current device's configuration exactly as an end-target SCSI device does. This serves two purposes: Improves security and reliability by not trusting the data supplied by remote initiator via SCSI low-level protocol. Some low-level SCSI protocols don't provide data transfer length and direction, so that information can be get only directly from CDB and current device's configuration. For example, for tape devices to get data transfer size it might be necessary to know block size setting. To process some exceptional conditions, like ILI on tape devices. To initialize incoming commands with some device-specific parameters, like timeout value. To allow some additional device-specific commands pre-, post- processing or alternative execution, like copying data from system cache, and do that completely independently from target drivers. Device handlers performs very lightweight processing and therefore should not considerably affect performance or CPU load. They are considered to be part of SCST, so they could directly access any fields in SCST's structures as well as use the corresponding functions. Without appropriate device handler SCST hides devices of this type from remote initiators and returns Device specific driver registration scst_register_dev_driver()

To work with SCST a device specific driver must register itself in SCST by calling int scst_register_dev_driver( struct scst_dev_type *dev_type) Where: The function returns 0 on success or appropriate error code otherwise. Structure Structure struct scst_dev_type { char name[15]; int type; unsigned parse_atomic:1; unsigned exec_atomic:1; unsigned dev_done_atomic:1; int (*init) (struct scst_dev_type *dev_type); void (*release) (struct scst_dev_type *dev_type); int (*attach) (struct scst_device *dev); void (*detach) (struct scst_device *dev); int (*attach_tgt) (struct scst_tgt_device *tgt_dev); void (*detach_tgt) (struct scst_tgt_device *tgt_dev); int (*parse) (struct scst_cmd *cmd); int (*exec) (struct scst_cmd *cmd, void (*scst_cmd_done)(struct scsi_cmnd *cmd, int next_state)); int (*dev_done) (struct scst_cmd *cmd); int (*task_mgmt_fn) (struct scst_mgmt_cmd *mgmt_cmd, struct scst_tgt_dev *tgt_dev, struct scst_cmd *cmd_to_abort); int (*on_free_cmd) (struct scst_cmd *cmd); int (*proc_info) (char *buffer, char **start, off_t offset, int length, int *eof, struct scst_dev_type *dev_type, int inout) struct module *module; } Where: bufflen/ and data_direction/ (see below If the driver needs to create additional files in its /proc subdirectory, it can use Structure struct scst_info_cdb { enum scst_cdb_flags flags; scst_data_direction direction; unsigned int transfer_len; unsigned short cdb_len; const char *op_name; } Where: Field data_direction/, set by Device specific driver unregistration

Device specific driver is unregistered by calling void scst_unregister_dev_driver( struct scst_dev_type *dev_type) Where: SCST commands' states

There are the following states, which a SCST command passes through during execution and which could be returned by device handler's tgt_dev/ assignment) state SCST's structures manipulation functions

Target drivers must not directly access any fields in SCST's structures, they must use only described below functions. SCST target driver manipulation functions scst_tgt_get_tgt_specific() and scst_tgt_set_tgt_specific()

Function void *scst_tgt_get_tgt_specific( struct scst_tgt *tgt) Function void scst_tgt_set_tgt_specific( struct scst_tgt *tgt, void *val) Where: SCST session manipulation functions scst_sess_get_tgt_specific() and scst_sess_set_tgt_specific()

Function void *scst_sess_get_tgt_specific( struct scst_session *sess) Function void scst_sess_set_tgt_specific( struct scst_session *sess, void *val) Where: SCST command manipulation functions scst_cmd_atomic()

Function int scst_cmd_atomic( struct scst_cmd *cmd) Where: scst_cmd_get_session()

Function struct scst_session *scst_cmd_get_session( struct scst_cmd *cmd) Where: scst_cmd_get_resp_data_len()

Function unsigned int scst_cmd_get_resp_data_len( struct scst_cmd *cmd) Where: scst_cmd_get_tgt_resp_flags()

Function int scst_cmd_get_tgt_resp_flags( struct scst_cmd *cmd) Where: scst_cmd_get_buffer()

Function void *scst_cmd_get_buffer( struct scst_cmd *cmd) Where: It is recommended to use scst_cmd_get_bufflen()

Function unsigned int scst_cmd_get_bufflen( struct scst_cmd *cmd) Where: It is recommended to use scst_cmd_get_use_sg()

Function unsigned short scst_cmd_get_use_sg( struct scst_cmd *cmd) Where: It is recommended to use scst_cmd_get_data_direction()

Function scst_data_direction scst_cmd_get_data_direction( struct scst_cmd *cmd) Where: scst_cmd_get_status()

Functions uint8_t scst_cmd_get_status( struct scst_cmd *cmd) Where: scst_cmd_get_masked_status()

Functions uint8_t scst_cmd_get_masked_status( struct scst_cmd *cmd) Where: scst_cmd_get_msg_status()

Functions uint8_t scst_cmd_get_msg_status( struct scst_cmd *cmd) Where: scst_cmd_get_host_status()

Functions uint16_t scst_cmd_get_host_status( struct scst_cmd *cmd) Where: scst_cmd_get_driver_status()

Functions uint16_t scst_cmd_get_driver_status( struct scst_cmd *cmd) Where: scst_cmd_get_sense_buffer()

Functions uint8_t *scst_cmd_get_sense_buffer( struct scst_cmd *cmd) Where: scst_cmd_get_sense_buffer_len()

Functions int scst_cmd_get_sense_buffer_len( struct scst_cmd *cmd) Where: scst_cmd_get_tag() and scst_cmd_set_tag()

Function uint32_t scst_cmd_get_tag( struct scst_cmd *cmd) Function void scst_cmd_set_tag( struct scst_cmd *cmd, uint32_t tag) Where: scst_cmd_get_tgt_specific() and scst_cmd_get_tgt_specific_lock()

Functions void *scst_cmd_get_tgt_specific( struct scst_cmd *cmd) void *scst_cmd_get_tgt_specific_lock( struct scst_cmd *cmd) Where: scst_cmd_set_tgt_specific() and scst_cmd_set_tgt_specific_lock()

Functions void *scst_cmd_set_tgt_specific( struct scst_cmd *cmd, void *val) void *scst_cmd_set_tgt_specific_lock( struct scst_cmd *cmd, void *val) Where: scst_cmd_get_data_buff_alloced() and scst_cmd_set_data_buff_alloced()

Function int scst_cmd_get_data_buff_alloced( struct scst_cmd *cmd) Function void scst_cmd_set_data_buff_alloced( struct scst_cmd *cmd) Where: scst_cmd_set_expected(), scst_cmd_is_expected_set(), scst_cmd_get_expected_data_direction() and scst_cmd_get_expected_transfer_len()

Function void scst_cmd_set_expected( struct scst_cmd *cmd, scst_data_direction expected_data_direction, unsigned int expected_transfer_len) Function int scst_cmd_is_expected_set( struct scst_cmd *cmd) Function scst_data_direction scst_cmd_get_expected_data_direction( struct scst_cmd *cmd) Function unsigned int scst_cmd_get_expected_transfer_len( struct scst_cmd *cmd) Where: scst_get_buf_first(), scst_get_buf_next(), scst_put_buf() and scst_get_buf_count()

These functions are designed to simplify and unify access to the commands data (SG vector or plain data buffer) in all possible conditions, including HIGHMEM environment, and should be used instead of direct access. Function int scst_get_buf_first( struct scst_cmd *cmd, uint8_t **buf) Where: Returns the length of the chunk of data for success, 0 for the end of data, negative error code otherwise. Function int scst_get_buf_next( struct scst_cmd *cmd, uint8_t **buf) Where: Returns the length of the chunk of data for success, 0 for the end of data, negative error code otherwise. Function void scst_put_buf( struct scst_cmd *cmd, uint8_t *buf) Where: Function int scst_get_buf_count( struct scst_cmd *cmd) Where: SCST task management commands manipulation functions scst_mgmt_cmd_get_tgt_specific()

Function void *scst_mgmt_cmd_get_tgt_specific( struct scst_mgmt_cmd *mcmd) Where: scst_mgmt_cmd_get_status()

Functions void *scst_mgmt_cmd_get_status( struct scst_mgmt_cmd *mcmd) Where: The following status values are possible: SCST_MGMT_STATUS_SUCCESS - the task management command completed successfully SCST_MGMT_STATUS_FAILED - the task management command failed. Miscellaneous functions scst_find_cmd_by_tag()

Function struct scst_cmd *scst_find_cmd_by_tag( struct scst_session *sess, uint32_t tag) Where: Returns found command or NULL otherwise. scst_find_cmd()

Function struct scst_cmd *scst_find_cmd( struct scst_session *sess, void *data, int (*cmp_fn)(struct scst_cmd *cmd, void *data)) Where: Returns found command or NULL otherwise. scst_get_cdb_info()

Function int scst_get_cdb_info( const uint8_t *cdb_p, int dev_type, struct scst_info_cdb *info_p) Where: Returns 0 on success, -1 otherwise. scst_to_dma_dir()

Function int scst_to_dma_dir( int scst_dir) Where: Returns the corresponding scst_is_cmd_local()

Function int scst_is_cmd_local( struct scst_cmd *cmd) Where: Returns 1, if the command's CDB is locally handled by SCST or 0 otherwise scst_register_virtual_device() and scst_unregister_virtual_device()

These functions provide a way for device handlers to register a virtual (emulated) device, which will be visible only by remote initiators. For example, FILEIO device handler uses files on file system to makes from them virtual remotely available SCSI disks. Function int scst_register_virtual_device( struct scst_dev_type *dev_handler) Where: Returns assigned to the device ID on success, or negative value otherwise. Function void scst_unregister_virtual_device( int id) Where: scst_add_threads() and scst_del_threads()

These functions allows to add or delete some SCST threads. For example, if int scst_add_threads( int num) Where: Returns 0 on success, error code otherwise. Function void scst_del_threads( int num) Where: scst_proc_get_tgt_root()

Function struct proc_dir_entry *scst_proc_get_tgt_root( struct scst_tgt_template *vtt) Where: Returns proc_dir_entry on success, NULL otherwise. scst_proc_get_dev_type_root()

Function struct proc_dir_entry *scst_proc_get_dev_type_root( struct scst_dev_type *dtt) Where: Returns proc_dir_entry on success, NULL otherwise.