The Linux SG_IO ioctl in the 2.6 series
2013-07-09 09:30
656 查看
http://gmd20.blog.163.com/blog/static/1684392320100227396270/
原文地址：http://sg.danny.cz/sg/sg_io.html
The Linux SG_IO ioctl in the 2.6 series
Introduction
SCSI and related command sets
SG_IO ioctl overview
SG_IO ioctl in the sg driver
SG_IO ioctl differences
open() considerations
SCSI command permissions
CAP_SYS_RAWIO from a user process
SG_IO and the st driver
Maximum transfer size per command
Conclusion
The SG_IO ioctl
permits user applications to send SCSI commands to a device. In the linux 2.4 series this ioctl was only
available via the SCSI generic (sg) driver. In the linux 2.6 series the SG_IO ioctl is additionally available for block devices and SCSI tape (st) devices. So there are multiple implementations of this ioctl within the kernel with slightly different characteristics
and describing these is the purpose of this document.
The information in this page is valid for linux kernel 2.6.16 .
All SCSI devices should respond to an INQUIRY command and part of their response is the so-called peripheral device type. This is
used by the linux kernel to decide which upper level driver controls the device. There are also devices that belong to other (i.e. not considered SCSI) transports that use SCSI command sets, the primary examples of this are (S-)ATAPI CD and DVD drives. Not
all peripheral device types map to upper level drivers and devices of these types are usually accessed via the SCSI generic (sg) driver.
SCSI (draft) standards are found at www.t10.org .
SCSI commands common to all SCSI devices are found in SPC-4 while those specific to block devices are found in SBC-2, those for CD/DVD drives are found in MMC-5 and those for SCSI tape drives are found in SSC-3.
The major non-SCSI command set in the storage area is for ATA non-packet devices
which are typically disks. ATApacket devices
use ATAPI which in the vast majority of cases carry a SCSI command set. The most recent draft ATA command set standard is ATA8-ACS and can be found at www.t13.org .
To complicate things (non-packet) ATA devices may have their native command set translated into SCSI. This can happen in the kernel (e.g. libata in linux) or in an intermediate device (e.g. in a USB external disk enclosure). Yet another possibility are disks
whose firmware can be changed to allow them to use either the SCSI or ATA command set, this may happen in the SAS/SATA area since the physical (cabling) and phy (electrical signalling) levels are so similar.
The third argument given to the SG_IO ioctl is a pointer to an instance of the sg_io_hdr structure which is defined in the <scsi/sg.h>
header file. The execution of the SG_IO ioctl can viewed as going through three phases:
do sanity checks on the metadata in the sg_io_hdr instance; read the input fields and the data pointed to by some of those fields; build a SCSI command and issue it to the device
wait for either a response from the device, the command to timeout or the user to terminate the process (or thread) that invoked the SG_IO ioctl
write the output fields and in some cases write data to locations pointed to by some fields, then return
Only
phase 1 returns an ioctl error (i.e. a return value of -1 and a value set in errno). In phase 2, command timeouts should be used sparingly as the device (and some others on the same interconnect) may end up being reset. If the user terminates the process or
thread that invoked the SG_IO ioctl then obviously phase 3 never occurs but the command execution runs to completion (or timeout) and the kernel "throws away" the results. If the command yields a SCSI status of CHECK CONDITION (in field "status") then sense
data
is written out in phase 3 .
Now we will assume that the SCSI command involves user data
being transferred to or from the device. The SCSI subsystem does not support true bidirectional data
transfers to a device. All data
DMA transfers (assuming the hardware supports DMA) occur in phase 2. However, if indirect IO is being used (i.e. neither direct IO nor mmap-ed transfers) then either:
data is read from the user space in phase 1 into kernel buffers and DMA-ed to the device in phase 2, or
data is read from the device into kernel buffers in phase 2 and written into the user space in phase 3
When direct IO or mmap-ed transfers are being used then all user data
is moved in phase 2 . If a process is terminated during such a data
transfer then the kernel gracefully handles this (by pinning the associated memory pages until the transfer is complete).
The sg_io_hdr structure has 22 fields (members) but typically only
a small number of them need to be set. The following code
fragment shows the setup for a simple TEST UNIT READY SCSI command which has no associated data
transfers:
unsigned char sense_b[32];
unsigned char turCmbBlk[] = {TUR_CMD, 0, 0, 0, 0, 0};
struct sg_io_hdr io_hdr;
memset(&io_hdr, 0, sizeof(struct sg_io_hdr));
io_hdr.interface_id = 'S';
io_hdr.cmd_len = sizeof(turCmbBlk);
io_hdr.mx_sb_len = sizeof(sense_b);
io_hdr.dxfer_direction = SG_DXFER_NONE;
io_hdr.cmdp = turCmbBlk;
io_hdr.sbp = sense_b;
io_hdr.timeout = DEF_TIMEOUT;
if (ioctl(fd, SG_IO, &io_hdr) < 0) {
The memset() call is pretty important,
setting unused input fields to safe values. Setting the timeout field to zero is not a good idea; 30,000 (for 30 seconds) is a reasonable default for most SCSI commands. As always, good error processing consumes a lot more code.
This is especially the case with SCSI commands that yield "sense data"
when something goes wrong. For example, if there is a medium error during a disk read, the sense data
will contain the logical block address (lba) of the failure. Another error processing example is a SCSI command that the device considers an "illegal request", the sense data
may show the byte and bit position of the field in the command block (usually referred to as a "cdb") that it objects to. For examples on error processing please refer to the sg3_utils package, its "examples" directory and its library components: sg_lib.c
(SCSI error processing and tables) and sg_cmds.c (common SCSI commands).
Below is a grouping of important
sg_io_hdr structure fields with brief summaries:
Command block (historically referred to as the "cdb"):
cmdp - pointer to cdb (the SCSI command block)
cmd_len - length (in bytes) of cdb
Data
transfer:
dxferp - pointer to user data to start reading from or start writing to
dxfer_len - number of bytes to transfer
dxfer_direction - whether to read from device (into user memory) or write to device (from user memory) or transfer no data: DXFER_FROM_DEV, DXFER_TO_DEV or DXFER_NONE respectively
resid - requested number of bytes to transfer (i.e. dxfer_len) less the actual number transferred
Error indication:
status - SCSI status returned from the device
host_status - error from Host Bus Adapter including initiator (port)
driver_status - driver (mid level or low level driver) error and suggestion mask
Sense data
(only
used when 'status' is CHECK CONDITION or (driver_status & DRIVER_SENSE) is true):
sbp - pointer to start writing sense data to
mx_sb_len - maximum number of bytes to write to sbp
sb_len_wr - actual number of bytes written to sbp
The fields in the sg_io_hdr structure are defined in more detail in the SCSI-Generic-HOWTO document.
Linux kernel 2.4.0 was the first production kernel in which the SG_IO ioctl appeared in the SCSI generic (sg) driver. The sg driver
itself has been in linux since around 1993. An instance of the sg_io_hdr structure in the sg driver can either be:
pointed to by the third argument of the SG_IO ioctl
pointed to by the second argument of UNIX write() or read() system calls which have a file descriptor of a sg device node as their first argument
The SCSI-Generic-HOWTO document
describes the sg driver in the lk 2.4 series including its use of the SG_IO ioctl. Prior to the lk 2.4 series the sg driver only
had the sg_header structure. It was used as an asynchronous command interface in which command, metadata and optionally user data
was sent via a Unix write() system call. The corresponding response which included error information (e.g. sense data)
or optionally user data
was received via a Unix read() system call. Two major additions were made to the sg driver at the beginning of the lk 2.4 series:
a new metadata structure (sg_io_hdr) as an alternative to the original mixed metadata and data structure (sg_header)
the SG_IO ioctl that used the new metadata structure and was synchronous: it sent a SCSI command and waited for its reply
The sg_io_hdr only
contains metadata in the sense that it contains pointers to locations of where data
will come from (command or data
in) or go to (sense data
or data
out). These pointers have caused problems in mixed 32/64 bit environments, especially when the user application (e.g. cdrecord) is built for 32 bits and the kernel is 64 bits. The lk 2.6 series has a compatibility layer to cope with this via code
specialized for the SG_IO ioctl. Unfortunately this problem was not foreseen when the sg_io_hdr structure was designed.
A significant feature of the SG_IO ioctl in the sg driver is that it is user interruptible. This means between issuing a command
(e.g. a long duration command like a disk format) and its response arriving a user could hit control-C on the associated application. The kernel would remain stable and resources would be cleared up at the appropriate time. The sg driver does not attempt to
abort such a command that is "in flight", it simply throws away the response and cleans up. Naturally the user has no direct way of finding out whether an interrupted command succeeded or not, by there may be indirect ways.
A warning may also be in order here: a long duration command such as format would typically be given a long timeout value. If the
user interrupted the application that sent the format command then the device may remain busy doing the format (especially if the IMMED bit is not set). So if the user then sent a short duration command such as TEST UNIT READY or REQUEST SENSE to see what
the device was doing, these commands may timeout. This would invoke the SCSI subsystem error handler which would most likely send a device reset, thus aborting the format, to get the device's attention. This is probably not what the user had in mind!
In the following table, sg_io_hdr structure fields are listed in the order they appear in that structure. Basically the "in" fields
appear at the top of the structure and are read in phase 1. The latter fields are termed as "out" and are written by the SG_IO implementation in phase 3.
Table 1. sg_io_hdr structure summary and implementation differences
The DID_* and DRIVER_* error and suggestion codes (associated with host_status and driver_status) are discussed in more detail in
the SCSI-Generic-HOWTO document.
Various drivers have different characteristics when a device node is opened. One
problem with the ioctl system call is that a user only
needs read permissions to execute it but may, with the ioctls like SG_IO, write to a device (e.g. format it). Command (operation code)
sniffing logic is used to overcome this security problem. Also users of the SG_IO ioctl need to be aware when they "share" a device with sd, st or a cdrom driver that state machines within those drivers may be tricked. This may be unavoidable but the users
of the SG_IO ioctl should take appropriate care.
Opening a file in linux with flags of zero implies the O_RDONLY flag and hence read only
access. All open() system calls can yield ENOENT (no such file or directory); ENODEV (no such device) if the file exists but there is no attached device and EACCES (permission denied) if the user doesn't have appropriate permissions.
A user with CAP_SYS_RAWIO capability (normally associated with the "root" user) bypasses all command sniffing and other access controls
that would otherwise lead to EACCES or EPERM errors. With the sg driver such a user may still need to open() a device node with O_RDWR (rather than O_RDONLY) to use all SCSI commands.
Table 2. open() flags for SG_IO ioctl usage
Notes:
on subsequent SG_IO ioctl calls, the sg driver will only allow SCSI commands in its allow_ops array, others result in EPERM (operation not permitted) in errno. See below .
if previous open() of this sg device node still holds O_EXCL then this open() waits until it clears.
on subsequent SG_IO ioctl calls, the block layer will only allow SCSI commands listed as "safe_for_read" in the verify_command() function in the drivers/block/scsi_ioctl.c file; others result in EPERM (operation not permitted) in errno. See below .
if removable media and it is not present then yields ENOMEDIUM (no medium found)
if a tape is not present in drive then yields EIO (input/output error), if tape is "in use" then yields EBUSY (resource busy). Only one open file descriptor is allowed per st device node at a time (although dup() can be used).
if tray closed and media is not present then yields ENOMEDIUM (no medium found); if tray open then tries to close it and if no media present then yields ENOMEDIUM
if previous open() of this sg device node still holds O_EXCL then yields EBUSY (resource busy).
on subsequent SG_IO ioctl calls, the block layer will allow SCSI commands listed as either "safe_for_read" or "safe_for_write". For other SCSI commands the user requires the CAP_SYS_RAWIO capability (usually associated with the "root" user); if not yields
EPERM (operation not permitted). The first instance of other SCSI commands since boot, sends an annoying "scsi: unknown opcode" message to the log.
if the media or drive is marked as not writable then yields EROFS (read-only file system).
if sg device node already has exclusive lock then a subsequent attempt to open(O_EXCL) will wait unless O_NONBLOCK is given in which case it yields EBUSY (resource busy)
implemented at block device level (which knows about partitions within devices). If a previous open(O_EXCL) is active then a subsequent open(O_EXCL) yields EBUSY (resource busy). Mounted file systems typically open a device/partition with O_EXCL; as long
as an application using the SG_IO ioctl does not also try and use the O_EXCL flag then it will be allowed access to the device.
the st driver does not support (i.e. ignores) the O_EXCL flag. However the fact that it only permits one active open() per tape device is similar functionality.
if tape is "in use" then yields EBUSY (resource busy). Only one open file descriptor is allowed per st device node at a time.
The O_EXCL flag has a different effect in the sg driver and the block layer. In the sg driver, once
O_EXCL is held on a device, all subsequent open() attempts will either wait or yield EBUSY (irrespective of whether they attempt to use the O_EXCL flag). Once
a partition/device is opened successfully in the block layer (with the sd or cdrom driver) only
subsequent open() attempts that also use the O_EXCL flag are rejected (with EBUSY). A O_EXCL lock held on a device in the block layer has no effect on accessing the same device via the sg driver (and vice versa).
The first successful open on a sd or a cdrom device node that has removable media will send a PREVENT ALLOW MEDIUM REMOVAL (prevent)
SCSI command to the device. If successful, this will inhibit a subsequent START STOP UNIT (eject) SCSI command and de-activate the eject button on the drive. In emergencies, the SG_IO ioctl can be used to defeat this action,
an example of this is the sdparm utility,
specifically "sdparm --command=unlock".
The open() flag O_NDELAY has the same value and meaning as O_NONBLOCK. Other flags such as O_DIRECT, O_TRUNC and O_APPEND have no
effect on the SG_IO ioctl.
In linux a user only
needs read permissions on a file descriptor to execute an ioctl() system command. In the case of the SG_IO ioctl, a SCSI command could be sent that obviously changes the state of a device (e.g. WRITE to a disk). So both implementations of the SG_IO ioctl require
more than read permissions for some commands, especially those that are known to change the state of a device or those that have some unknown action
(e.g. vendor specific commands).
Here is a table of SCSI commands that don't need the user to have write permissions (or in some cases CAP_SYS_RAWIO capability which
usually equates to "root" user):
Table 3. SCSI command minimum permission requirements
Any other SCSI command (opcode) not mentioned for the sg driver needs O_RDWR. Any other SCSI command (opcode) not mentioned for
the block layer SG_IO ioctl needs a user with CAP_SYS_RAWIO capability. All "block" SG_IO ioctl calls on st device nodes need a user with CAP_SYS_RAWIO capability. If a user does not have sufficient permissions to execute a SCSI command via the SG_IO ioctl
then the system calls fails (i.e. no SCSI command is sent) and errno is set to EPERM (operation not permitted).
Both the sg driver and the block layer SG_IO code
use internal tables to enforce the permissions shown in the above table (allow_ops and cmd_type [safe_for_read and safe_for_write] respectively). This technique doesn't scale well, since more advanced command sets (e.g. OSD) use service actions (and one
opcode: 0x7f in the case of OSD). There may also be overlap in opcode usage between command sets, for example between SBC, MMC and SSC.
While root processes usually have CAP_SYS_RAWIO, processes running under a user's ID (i.e. non-root) typically don't. Hence non-root
processes may not be able to use SG_IO to send SCSI commands that require CAP_SYS_RAWIO. This may occur even if the permission bits of the device node file allow for read or write access, user processes will receive EPERM when using SG_IO.
By default the capability to assign capabilities to other processes (CAP_SETPCAP) is limited to very few processes, such as certain
kernel threads. Changing this default would require to change and recompile the kernel.
Processes which are forked by a root process and call setuid later will lose the CAP_SYS_RAWIO capability the parent root process
(and the child before the setuid) had. However, the child can preserve the capabilities of the root process in the permitted set and raise it after the call of setuid:
/* ... in child after fork(), still running as root ... */
prctl(PR_SET_KEEPCAPS, 1, 0, 0, 0);
setuid(...);
cap_set_proc(cap_from_text("cap_sys_rawio+ep"));
This way a user process with a parent root process can 'get back' the required capabilities to directly send SCSI commands to a
device via SG_IO.
The above technique may be of use to daemons that are started with root permissions (most are) and then changes to another user
after a fork(). It is not obvious to the author how utilities that use the SG_IO ioctl on device nodes that require CAP_SYS_RAWIO for some or all SCSI commands (e.g. nodes associated with the sd and st drivers) can use the above technique.
In order to implement its user space API, the st driver has to maintain information about where the read head is with respect to
the structural elements of the tape (filemarks, beginning of tape, end of data).
Because the streaming device SCSI commands don't have addresses, the st driver has to know what commands have been sent. When reading, the filemarks are noticed when a read fails and sense data
is fethed. If SG_IO is mixed with tape commands, the st driver may lose information (it does not look at the SG_IO commands and results). Because of this, the st driver may not implement the semantics the user expects. If the user accepts this or knows when
using SG_IO does not cause information loss, then using SG_IO is OK.
So mixing st driver read, write and ioctl commands with SCSI commands sent via SG_IO that change the state of the tape is not recommended.
This applies whether the SG_IO SCSI commands are sent via st or sg device nodes.
The largest amount of data
that can be transferred by a single SCSI command is often a concern. Various SCSI command sets (e.g. SBC-3 for disk READs and WRITEs, SSC-3 for tape READs and WRITEs, and SPC-4 for READ+WRITE BUFFER) allow very large data
transfer sizes but Linux is not so accommodating. The Host Bus Adapter (HBA) could have transfer size limits as could the transport and finally the SCSI device itself. In the latter case SBC-3 defines a "Block Limits" Vital Product Data
(VPD) while SSC has the READ BLOCK LIMITS SCSI command. SBC-3's optional Block Limits VPD page contains both maximum and optimal counts. In the author's opinion that latter distinction is very important:
the block susbsystem should try and use optimal sizes while pass through users should only
be constrained by maximum sizes. Also if a pass through user exceeds a maximum transfer size imposed by a SCSI device, then the device can report an error. There is an underlying assumption that the applications using a pass through interface know what they
are doing, or at least know more than the various kernel susbsystems. On the other hand, the kernel has the responsibility to allocate critical shared resources such as memory.
In the past, Linux used a single, "big-enough", block of memory for the source or destination of large data
transfers. Then scatter-gather lists where added to break transfers up into smaller (often "page" size (4 KB on i386 architecture)) chunks which made memory management easier for the kernel. Now, in the lk 2.6 series, the single block of memory option is being
phased out.
The Linux SCSI subsystem imposes a 128 element limit on scatter gather lists via its SCSI_MAX_PHYS_SEGMENTS define. The way various
memory pools are allocated by the linux SCSI subsystem, SCSI_MAX_PHYS_SEGMENTS could be increased to 256. Associated with each type of HBA there is normally a low level driver (LLD). Each LLD can further limit the maximum number of elements with the scsi_host_template::sg_tablesize
field. Prior to lk 2.6.16 the sg and st drivers used the .sg_tablesize field only,
since lk 2.6.16 those drivers are also constrained by SCSI_MAX_PHYS_SEGMENTS. This leads to a potential halving of the maximum transfer size. Many LLDs set the .sg_tablesize field to SG_ALL (which is 255) but they may as well set that field to 256 unless the
HBA hardware has a constraint.
User space memory may be allocated as the source and/or destination for DMA transfers from the HBA (i.e. direct IO). Even if the
user space allocated a large amount of memory with a single malloc(), the HBA DMA element typically has a different view of memory. This view may well contain many "page" size discontinuous pieces. This has the effect of using up, or perhaps exhausting, scatter-gather
elements.
The sg driver attempts to build scatter gather lists with each element up to SG_SCATTER_SZ bytes large. This define is found in
include/scsi/sg.h and has been set to 32 KB for some years. That is 8 times the page size (of 4 KB) on the i386 architecture. Some users who need really large transfers increase this define (and it is best to keep it a power of 2). However since lk 2.6.16
another limit comes into play: the MAX_SEGMENT_SIZE define which is set to 64 KB. MAX_SEGMENT_SIZE is a default and can be overridden by the LLD calling blk_queue_max_segment_size().
In lk 2.6.16 two further LLD parameters come into play even when the sg (and st) driver is used. These are scsi_host_template::max_sectors
and scsi_host_template::use_clustering .
The .max_sectors setting in the LLD is the maximum number of 512 byte sectors allowed in a single SCSI command's scatter gather
lists (for data
transfers). Yes, that is a strange limit when trying to send a SCSI WRITE BUFFER command to upload firmware. Sysfs makes the LLD's .max_sectors setting visible (converted to kilobytes) in /sys/block/sd<x>/queue/max_hw_sectors_kb . The maximum allowable value
in a LLD's .max_sector seems to be 65535 (0xffff in hexadecimal). This limits the maximum transfer size to (32*1024*1024 - 512) bytes, assuming other limitations have been overcome. [The 65535 sector limit is because Scsi_Host::max_sectors has type "unsigned
short". Hopefully this type is expanded to "int" in the future (or removed).]
The .use_clustering field should be set to ENABLE_CLUSTERING . If not, the block subsystem rebuilds the scatter gather list it gets
from the sg driver with page size (e.g. 4 KB) elements. [Actually is does that anyway, but when ENABLE_CLUSTERING is set, it coalesces them again!]
In some situations, sending commands via the SG_IO ioctl may interfere with a higher level driver's use of a device. Users of the
SG_IO ioctl should be aware that they are using a powerful, but low level facility, and write code
accordingly. An example of this would be a utility to perform self tests on a disk: "background" self tests should be preferred over "foreground" self tests if there is a chance the computer may be using a file system on that disk at the time. Even a short
foreground self test may take up to two minutes which is a long time to lock out a file system.
Return to main page.
Last updated: 26th July 2008
原文地址：http://sg.danny.cz/sg/sg_io.html
The Linux SG_IO ioctl in the 2.6 series
The Linux SG_IO ioctl in the 2.6 seriesIntroduction
SCSI and related command sets
SG_IO ioctl overview
SG_IO ioctl in the sg driver
SG_IO ioctl differences
open() considerations
SCSI command permissions
CAP_SYS_RAWIO from a user process
SG_IO and the st driver
Maximum transfer size per command
Conclusion
Introduction
The SG_IO ioctlpermits user applications to send SCSI commands to a device. In the linux 2.4 series this ioctl was only
available via the SCSI generic (sg) driver. In the linux 2.6 series the SG_IO ioctl is additionally available for block devices and SCSI tape (st) devices. So there are multiple implementations of this ioctl within the kernel with slightly different characteristics
and describing these is the purpose of this document.
The information in this page is valid for linux kernel 2.6.16 .
SCSI and related command sets
All SCSI devices should respond to an INQUIRY command and part of their response is the so-called peripheral device type. This isused by the linux kernel to decide which upper level driver controls the device. There are also devices that belong to other (i.e. not considered SCSI) transports that use SCSI command sets, the primary examples of this are (S-)ATAPI CD and DVD drives. Not
all peripheral device types map to upper level drivers and devices of these types are usually accessed via the SCSI generic (sg) driver.
SCSI (draft) standards are found at www.t10.org .
SCSI commands common to all SCSI devices are found in SPC-4 while those specific to block devices are found in SBC-2, those for CD/DVD drives are found in MMC-5 and those for SCSI tape drives are found in SSC-3.
The major non-SCSI command set in the storage area is for ATA non-packet devices
which are typically disks. ATApacket devices
use ATAPI which in the vast majority of cases carry a SCSI command set. The most recent draft ATA command set standard is ATA8-ACS and can be found at www.t13.org .
To complicate things (non-packet) ATA devices may have their native command set translated into SCSI. This can happen in the kernel (e.g. libata in linux) or in an intermediate device (e.g. in a USB external disk enclosure). Yet another possibility are disks
whose firmware can be changed to allow them to use either the SCSI or ATA command set, this may happen in the SAS/SATA area since the physical (cabling) and phy (electrical signalling) levels are so similar.
SG_IO ioctl overview
The third argument given to the SG_IO ioctl is a pointer to an instance of the sg_io_hdr structure which is defined in the <scsi/sg.h>header file. The execution of the SG_IO ioctl can viewed as going through three phases:
do sanity checks on the metadata in the sg_io_hdr instance; read the input fields and the data pointed to by some of those fields; build a SCSI command and issue it to the device
wait for either a response from the device, the command to timeout or the user to terminate the process (or thread) that invoked the SG_IO ioctl
write the output fields and in some cases write data to locations pointed to by some fields, then return
Only
phase 1 returns an ioctl error (i.e. a return value of -1 and a value set in errno). In phase 2, command timeouts should be used sparingly as the device (and some others on the same interconnect) may end up being reset. If the user terminates the process or
thread that invoked the SG_IO ioctl then obviously phase 3 never occurs but the command execution runs to completion (or timeout) and the kernel "throws away" the results. If the command yields a SCSI status of CHECK CONDITION (in field "status") then sense
data
is written out in phase 3 .
Now we will assume that the SCSI command involves user data
being transferred to or from the device. The SCSI subsystem does not support true bidirectional data
transfers to a device. All data
DMA transfers (assuming the hardware supports DMA) occur in phase 2. However, if indirect IO is being used (i.e. neither direct IO nor mmap-ed transfers) then either:
data is read from the user space in phase 1 into kernel buffers and DMA-ed to the device in phase 2, or
data is read from the device into kernel buffers in phase 2 and written into the user space in phase 3
When direct IO or mmap-ed transfers are being used then all user data
is moved in phase 2 . If a process is terminated during such a data
transfer then the kernel gracefully handles this (by pinning the associated memory pages until the transfer is complete).
The sg_io_hdr structure has 22 fields (members) but typically only
a small number of them need to be set. The following code
fragment shows the setup for a simple TEST UNIT READY SCSI command which has no associated data
transfers:
unsigned char sense_b[32];
unsigned char turCmbBlk[] = {TUR_CMD, 0, 0, 0, 0, 0};
struct sg_io_hdr io_hdr;
memset(&io_hdr, 0, sizeof(struct sg_io_hdr));
io_hdr.interface_id = 'S';
io_hdr.cmd_len = sizeof(turCmbBlk);
io_hdr.mx_sb_len = sizeof(sense_b);
io_hdr.dxfer_direction = SG_DXFER_NONE;
io_hdr.cmdp = turCmbBlk;
io_hdr.sbp = sense_b;
io_hdr.timeout = DEF_TIMEOUT;
if (ioctl(fd, SG_IO, &io_hdr) < 0) {
The memset() call is pretty important,
setting unused input fields to safe values. Setting the timeout field to zero is not a good idea; 30,000 (for 30 seconds) is a reasonable default for most SCSI commands. As always, good error processing consumes a lot more code.
This is especially the case with SCSI commands that yield "sense data"
when something goes wrong. For example, if there is a medium error during a disk read, the sense data
will contain the logical block address (lba) of the failure. Another error processing example is a SCSI command that the device considers an "illegal request", the sense data
may show the byte and bit position of the field in the command block (usually referred to as a "cdb") that it objects to. For examples on error processing please refer to the sg3_utils package, its "examples" directory and its library components: sg_lib.c
(SCSI error processing and tables) and sg_cmds.c (common SCSI commands).
Below is a grouping of important
sg_io_hdr structure fields with brief summaries:
Command block (historically referred to as the "cdb"):
cmdp - pointer to cdb (the SCSI command block)
cmd_len - length (in bytes) of cdb
Data
transfer:
dxferp - pointer to user data to start reading from or start writing to
dxfer_len - number of bytes to transfer
dxfer_direction - whether to read from device (into user memory) or write to device (from user memory) or transfer no data: DXFER_FROM_DEV, DXFER_TO_DEV or DXFER_NONE respectively
resid - requested number of bytes to transfer (i.e. dxfer_len) less the actual number transferred
Error indication:
status - SCSI status returned from the device
host_status - error from Host Bus Adapter including initiator (port)
driver_status - driver (mid level or low level driver) error and suggestion mask
Sense data
(only
used when 'status' is CHECK CONDITION or (driver_status & DRIVER_SENSE) is true):
sbp - pointer to start writing sense data to
mx_sb_len - maximum number of bytes to write to sbp
sb_len_wr - actual number of bytes written to sbp
The fields in the sg_io_hdr structure are defined in more detail in the SCSI-Generic-HOWTO document.
SG_IO ioctl in the sg driver
Linux kernel 2.4.0 was the first production kernel in which the SG_IO ioctl appeared in the SCSI generic (sg) driver. The sg driveritself has been in linux since around 1993. An instance of the sg_io_hdr structure in the sg driver can either be:
pointed to by the third argument of the SG_IO ioctl
pointed to by the second argument of UNIX write() or read() system calls which have a file descriptor of a sg device node as their first argument
The SCSI-Generic-HOWTO document
describes the sg driver in the lk 2.4 series including its use of the SG_IO ioctl. Prior to the lk 2.4 series the sg driver only
had the sg_header structure. It was used as an asynchronous command interface in which command, metadata and optionally user data
was sent via a Unix write() system call. The corresponding response which included error information (e.g. sense data)
or optionally user data
was received via a Unix read() system call. Two major additions were made to the sg driver at the beginning of the lk 2.4 series:
a new metadata structure (sg_io_hdr) as an alternative to the original mixed metadata and data structure (sg_header)
the SG_IO ioctl that used the new metadata structure and was synchronous: it sent a SCSI command and waited for its reply
The sg_io_hdr only
contains metadata in the sense that it contains pointers to locations of where data
will come from (command or data
in) or go to (sense data
or data
out). These pointers have caused problems in mixed 32/64 bit environments, especially when the user application (e.g. cdrecord) is built for 32 bits and the kernel is 64 bits. The lk 2.6 series has a compatibility layer to cope with this via code
specialized for the SG_IO ioctl. Unfortunately this problem was not foreseen when the sg_io_hdr structure was designed.
A significant feature of the SG_IO ioctl in the sg driver is that it is user interruptible. This means between issuing a command
(e.g. a long duration command like a disk format) and its response arriving a user could hit control-C on the associated application. The kernel would remain stable and resources would be cleared up at the appropriate time. The sg driver does not attempt to
abort such a command that is "in flight", it simply throws away the response and cleans up. Naturally the user has no direct way of finding out whether an interrupted command succeeded or not, by there may be indirect ways.
A warning may also be in order here: a long duration command such as format would typically be given a long timeout value. If the
user interrupted the application that sent the format command then the device may remain busy doing the format (especially if the IMMED bit is not set). So if the user then sent a short duration command such as TEST UNIT READY or REQUEST SENSE to see what
the device was doing, these commands may timeout. This would invoke the SCSI subsystem error handler which would most likely send a device reset, thus aborting the format, to get the device's attention. This is probably not what the user had in mind!
SG_IO ioctl differences
In the following table, sg_io_hdr structure fields are listed in the order they appear in that structure. Basically the "in" fieldsappear at the top of the structure and are read in phase 1. The latter fields are termed as "out" and are written by the SG_IO implementation in phase 3.
| sg_io_hdr field | in or out | type | different | brief description including differences between implementations |
| interface_id | in | int | guard field. Current implementations only accept " (int)'S' ". If not set, the sg driver sets errno to ENOSYS while the block layer sets it to EINVAL | |
| dxfer_direction | in | (-ve) int | minor | direction of data transfer. SG_DXFER_NONE and friends are defined as negative integers so the sg driver can discriminate between sg_io_hdr instances and those of sg_header. This nuance is irrelevant to non-sg driver usage of SG_IO. See below. |
| cmd_len | in | unsigned char | limits command length to 255 bytes. No SCSI commands (even variable length ones in OSD) are this long (yet) | |
| max_sb_len | in | unsigned char | maximum number of bytes of sense data that the driver can output via the sbp pointer | |
| iovec_count | in | unsigned short | yes | if not sg driver and greater than zero then the SG_IO ioctl fails with errno set to EOPNOTSUPP; sg driver treats dxferp as a pointer to an array struct sg_iovec when this field is greater than zero |
| dxfer_len | in | unsigned int | minor | number of bytes of data to transfer to or from the device. Upper limit for block devices related to/sys/block/<device>/queue/max_sectors_kb |
| dxferp | in [*in or *out] | void * | minor | pointer to (user space) data to transfer to (if reading from device) or transfer from (if writing to device). Further level of indirection in the sg driver when iovec_count is greater than 0 . |
| cmdp | in [*in] | unsigned char * | pointer to SCSI command. The SG_IO ioctl in the sg drive fails with errno set to EMSGSIZE if cmdp is NULL and EFAULT if it is invalid; the block layer sets errno to EFAULT in both cases. | |
| sbp | in [*out] | unsigned char * | pointer to user data area where no more than max_sb_len bytes of sense data from the device will be written if the SCSI status is CHECK CONDITION. | |
| timeout | in | unsigned int | yes (if = 0) | time in milliseconds that the SCSI mid-level will wait for a response. If that timer expires before the command finishes, then the command may be aborted, the device (and maybe others on the same interconnect) may be reset depending on error handler settings. Dangerous stuff, the SG_IO ioctl has no control (through this interface) of exactly what happens. In the sg driver a timeout value of 0 means 0 milliseconds, in the block layer (currently) it means 60 seconds. |
| flags | in | unsigned int | yes | Block layer SG_IO ioctl ignores this field; the sg driver uses it to request special services like direct IO or mmap-ed transfers. It is a bit mask. |
| pack_id | in -> out | int | unused (for user space program tag) | |
| usr_ptr | in -> out | void * | unused (for user space pointer tag) | |
| status | out | unsigned char | SCSI command status, zero implies GOOD | |
| masked_status | out | unsigned char | Logically: masked_status == ((status & 0x3e) >> 1). Old linux SCSI subsystem usage, deprecated. | |
| msg_status | out | unsigned char | SCSI parallel interface (SPI) message status (very old, deprecated) | |
| sb_len_wr | out | unsigned char | actual length of sense data (in bytes) output via sbp pointer. | |
| host_status | out | unsigned short | error reported by the initiator (port). These are the "DID_*" error codes in scsi.h | |
| driver_status | out | unsigned short | bit mask: error and suggestion reported by the low level driver (LLD). These are the "DRIVER_*" error codes in scsi.h | |
| resid | out | int | (dxfer_len - number_of_bytes_actually_transferred). Typically only set when there is a shortened DMA transfer from the device. Not necessarily an error. Older LLDs always yield zero. | |
| duration | out | unsigned int | number of milliseconds that elapsed between when the command was injected into the SCSI mid level and the corresponding "done" callback was invoked. Roughly the duration of the SCSI command in milliseconds. | |
| info | out | unsigned int | minor | bit mask indicating what was done (or not) and whether any error was detected. Block layer SG_IO ioctl only sets SG_INFO_CHECK if an error was detected |
the SCSI-Generic-HOWTO document.
open() considerations
Various drivers have different characteristics when a device node is opened. Oneproblem with the ioctl system call is that a user only
needs read permissions to execute it but may, with the ioctls like SG_IO, write to a device (e.g. format it). Command (operation code)
sniffing logic is used to overcome this security problem. Also users of the SG_IO ioctl need to be aware when they "share" a device with sd, st or a cdrom driver that state machines within those drivers may be tricked. This may be unavoidable but the users
of the SG_IO ioctl should take appropriate care.
Opening a file in linux with flags of zero implies the O_RDONLY flag and hence read only
access. All open() system calls can yield ENOENT (no such file or directory); ENODEV (no such device) if the file exists but there is no attached device and EACCES (permission denied) if the user doesn't have appropriate permissions.
A user with CAP_SYS_RAWIO capability (normally associated with the "root" user) bypasses all command sniffing and other access controls
that would otherwise lead to EACCES or EPERM errors. With the sg driver such a user may still need to open() a device node with O_RDWR (rather than O_RDONLY) to use all SCSI commands.
| open() flags | sg notes | sd notes | st notes | cdrom notes | Comments |
| <none> or O_RDONLY | 1, 2 | 3,4 | 3,5 | 3,6 | best to add O_NONBLOCK. For a device with removable media (e.g. tape drive) that depends on whether the drive or its media is being accessed. |
| O_RDONLY | O_NONBLOCK | 1,7 | 3 | 3,13 | 3 | recommended when SCSI commands are recognized as reading information from the device |
| O_RDWR | 2 | 4,8,9 | 5,8,9 | 6,8,9 | again, could be better to add O_NONBLOCK |
| O_RDWR | O_NONBLOCK | 7 | 8,9 | 8,9,13 | 8,9 | recommended when arbitrary (including vendor specific) SCSI commands are to be sent |
| << interaction with O_EXCL>> | 10 | 11 | 12 | 11 | only use when sure that no other application may want to access the device (or partition). A surprising number of applications do "poke around" devices. |
| << interaction with O_DIRECT>> | - | --> | - | --> | requires sector alignment on data transfers (ignored by sg and st) |
on subsequent SG_IO ioctl calls, the sg driver will only allow SCSI commands in its allow_ops array, others result in EPERM (operation not permitted) in errno. See below .
if previous open() of this sg device node still holds O_EXCL then this open() waits until it clears.
on subsequent SG_IO ioctl calls, the block layer will only allow SCSI commands listed as "safe_for_read" in the verify_command() function in the drivers/block/scsi_ioctl.c file; others result in EPERM (operation not permitted) in errno. See below .
if removable media and it is not present then yields ENOMEDIUM (no medium found)
if a tape is not present in drive then yields EIO (input/output error), if tape is "in use" then yields EBUSY (resource busy). Only one open file descriptor is allowed per st device node at a time (although dup() can be used).
if tray closed and media is not present then yields ENOMEDIUM (no medium found); if tray open then tries to close it and if no media present then yields ENOMEDIUM
if previous open() of this sg device node still holds O_EXCL then yields EBUSY (resource busy).
on subsequent SG_IO ioctl calls, the block layer will allow SCSI commands listed as either "safe_for_read" or "safe_for_write". For other SCSI commands the user requires the CAP_SYS_RAWIO capability (usually associated with the "root" user); if not yields
EPERM (operation not permitted). The first instance of other SCSI commands since boot, sends an annoying "scsi: unknown opcode" message to the log.
if the media or drive is marked as not writable then yields EROFS (read-only file system).
if sg device node already has exclusive lock then a subsequent attempt to open(O_EXCL) will wait unless O_NONBLOCK is given in which case it yields EBUSY (resource busy)
implemented at block device level (which knows about partitions within devices). If a previous open(O_EXCL) is active then a subsequent open(O_EXCL) yields EBUSY (resource busy). Mounted file systems typically open a device/partition with O_EXCL; as long
as an application using the SG_IO ioctl does not also try and use the O_EXCL flag then it will be allowed access to the device.
the st driver does not support (i.e. ignores) the O_EXCL flag. However the fact that it only permits one active open() per tape device is similar functionality.
if tape is "in use" then yields EBUSY (resource busy). Only one open file descriptor is allowed per st device node at a time.
The O_EXCL flag has a different effect in the sg driver and the block layer. In the sg driver, once
O_EXCL is held on a device, all subsequent open() attempts will either wait or yield EBUSY (irrespective of whether they attempt to use the O_EXCL flag). Once
a partition/device is opened successfully in the block layer (with the sd or cdrom driver) only
subsequent open() attempts that also use the O_EXCL flag are rejected (with EBUSY). A O_EXCL lock held on a device in the block layer has no effect on accessing the same device via the sg driver (and vice versa).
The first successful open on a sd or a cdrom device node that has removable media will send a PREVENT ALLOW MEDIUM REMOVAL (prevent)
SCSI command to the device. If successful, this will inhibit a subsequent START STOP UNIT (eject) SCSI command and de-activate the eject button on the drive. In emergencies, the SG_IO ioctl can be used to defeat this action,
an example of this is the sdparm utility,
specifically "sdparm --command=unlock".
The open() flag O_NDELAY has the same value and meaning as O_NONBLOCK. Other flags such as O_DIRECT, O_TRUNC and O_APPEND have no
effect on the SG_IO ioctl.
SCSI command permissions
In linux a user onlyneeds read permissions on a file descriptor to execute an ioctl() system command. In the case of the SG_IO ioctl, a SCSI command could be sent that obviously changes the state of a device (e.g. WRITE to a disk). So both implementations of the SG_IO ioctl require
more than read permissions for some commands, especially those that are known to change the state of a device or those that have some unknown action
(e.g. vendor specific commands).
Here is a table of SCSI commands that don't need the user to have write permissions (or in some cases CAP_SYS_RAWIO capability which
usually equates to "root" user):
| SCSI command | (draft) standard | sg driver requires | block layer SG_IO requires (except st) | Comments |
| BLANK | MMC-4 | O_RDWR | O_RDWR | |
| CLOSE TRACK/SESSION | MMC-4 | O_RDWR | O_RDWR | |
| ERASE | MMC-4 | O_RDWR | O_RDWR | |
| FLUSH CACHE | SBC-3, MMC-4 | O_RDWR | O_RDWR | Really SYNCHRONIZE CACHE command |
| FORMAT UNIT | SBC-3, MMC-4 | O_RDWR | O_RDWR | default command timeout may not be long enough |
| GET CONFIGURATION | MMC-4 | O_RDWR | O_RDONLY | reads CD/DVD metadata |
| GET EVENT STATUS NOTIFICATION | MMC-4 | O_RDWR | O_RDONLY | |
| GET PERFORMANCE | MMC-4 | O_RDWR | O_RDONLY | |
| INQUIRY | SPC-4 | O_RDONLY | O_RDONLY | All SCSI devices should respond to this command |
| LOAD UNLOAD MEDIUM | MMC-4 | O_RDWR | O_RDWR | MEDIUM may be replaced by CD, DVD or nothing |
| LOG SELECT | SPC-4 | O_RDWR | O_RDWR | used to change logging or clear logged data |
| LOG SENSE | SPC-4 | O_RDONLY | O_RDONLY | used to fetch logged data |
| MAINTENANCE COMMAND IN | SPC-4 | O_RDONLY | CAP_SYS_RAWIO | various "REPORT ..." commands such as REPORT SUPPORTED OPERATION CODES in here |
| MODE SELECT (6+10) | SPC-4 | O_RDWR | O_RDWR | Used to change SCSI device metadata |
| MODE SENSE (6+10) | SPC-4 | O_RDONLY | O_RDONLY | Used to read SCSI device metadata |
| PAUSE RESUME | MMC-4 | O_RDWR | O_RDONLY | |
| PLAY AUDIO (10) | MMC-4 | O_RDWR | O_RDONLY | |
| PLAY AUDIO MSF | MMC-4 | O_RDWR | O_RDONLY | |
| PLAY AUDIO TI | ?? | O_RDWR | O_RDONLY | opcode 0x48, unassigned to any spec in SPC-4 |
| PLAY CD | MMC-2 | O_RDWR | O_RDONLY | old, now SPARE IN in SPC-4 |
| PREVENT ALLOW MEDIUM REMOVAL | SPC-4, MMC-4 | O_RDWR | O_RDWR | sd, st and cdrom drivers use this internally |
| READ (6+10+12+16) | SBC-3 | O_RDONLY | O_RDONLY | READ(16) requires O_RDWR with the sg driver before lk2.6.11 |
| READ BUFFER | SPC-4 | O_RDONLY | O_RDONLY | |
| READ BUFFER CAPACITY | MMC-4 | O_RDWR | O_RDONLY | |
| READ CAPACITY(10) | SBC-3, MMC-4 | O_RDONLY | O_RDONLY | |
| READ CAPACITY(16) | SBC-3, MMC-4 | O_RDONLY | CAP_SYS_RAWIO | within SERVICE ACTION IN command. Needed for RAIDs larger than 2 TB |
| READ CD | MMC-4 | O_RDWR | O_RDONLY | |
| READ CD MSF | MMC-4 | O_RDWR | O_RDONLY | |
| READ CDVD CAPACITY | SBC-3, MMC-4 | O_RDONLY | O_RDONLY | Strange (old ?) name from cdrom.h . Actually is READ CAPACITY. |
| READ DEFECT (10) | SBC-3 | O_RDWR | O_RDONLY | |
| READ DISC INFO | MMC-4 | O_RDWR | O_RDONLY | |
| READ DVD STRUCTURE | MMC-4 | O_RDWR | O_RDONLY | |
| READ FORMAT CAPACITIES | MMC-4 | O_RDWR | O_RDONLY | |
| READ HEADER | MMC-2 | O_RDWR | O_RDONLY | |
| READ LONG (10) | SBC-3 | O_RDONLY | O_RDONLY | but not READ LONG (16) |
| READ SUB-CHANNEL | MMC-4 | O_RDWR | O_RDONLY | |
| READ TOC/PMA/ATIP | MMC-4 | O_RDWR | O_RDONLY | |
| READ TRACK (RZONE) INFO | MMC-4 | O_RDWR | O_RDONLY | In MMC-4 called READ TRACK INFO |
| RECEIVE DIAGNOSTIC | SPC-4 | O_RDONLY | CAP_SYS_RAWIO | the SES command set uses this command a lot. An SES device is only accessible via an sg device node |
| REPAIR (RZONE) TRACK | MMC-4 | O_RDWR | O_RDWR | |
| REPORT KEY | MMC-4 | O_RDWR | O_RDONLY | |
| REPORT LUNS | SPC-4 | O_RDONLY | CAP_SYS_RAWIO | mandatory since SPC-3 |
| REQUEST SENSE | SPC-4 | O_RDONLY | O_RDONLY | has uses other than those displaced by autosense |
| RESERVE (RZONE) TRACK | MMC-4 | O_RDWR | O_RDWR | |
| SCAN | MMC-4 | O_RDWR | O_RDONLY | |
| SEEK | MMC-4 | O_RDWR | O_RDONLY | |
| SEND CUE SHEET | MMC-4 | O_RDWR | O_RDWR | |
| SEND DVD STRUCTURE | MMC-4 | O_RDWR | O_RDWR | |
| [SEND EVENT] | MMC-2 | O_RDWR | cdrom.h associates opcode 0xa2 but MMC-2 uses opcode 0x5d ?? | |
| SEND KEY | MMC-4 | O_RDWR | O_RDWR | |
| SEND OPC INFORMATION | MMC-4 | O_RDWR | O_RDWR | |
| SERVICE ACTION IN | SPC-4, SBC-3 | O_RDONLY | CAP_SYS_RAWIO | READ CAPACITY (16) service action in here |
| SET CD SPEED | MMC-4 | O_RDWR | O_RDWR | cdrom.h calls this SET SPEED |
| SET STREAMING | MMC-4 | O_RDWR | O_RDWR | |
| START STOP UNIT | SBC-3, MMC-4 | O_RDWR | O_RDONLY | hmm |
| STOP PLAY/SCAN | MMC-4 | O_RDWR | O_RDONLY | |
| SYNCHRONIZE CACHE | SBC-3, MMC-4 | O_RDWR | O_RDWR | cdrom.h calls this FLUSH CACHE |
| TEST UNIT READY | SPC-4 | O_RDONLY | O_RDONLY | All SCSI devices should respond to this command |
| VERIFY (10+16) | SBC-3, MMC-4 | O_RDWR | O_RDONLY | |
| WRITE (6+10+12+16) | SBC-3 | O_RDWR | O_RDWR | |
| WRITE LONG (10+16) | SBC-3 | O_RDWR | O_RDWR | |
| WRITE VERIFY (10+16) | SBC-3, MMC-4 | O_RDWR | O_RDWR | only WRITE VERIFY(10) is in MMC-4 |
the block layer SG_IO ioctl needs a user with CAP_SYS_RAWIO capability. All "block" SG_IO ioctl calls on st device nodes need a user with CAP_SYS_RAWIO capability. If a user does not have sufficient permissions to execute a SCSI command via the SG_IO ioctl
then the system calls fails (i.e. no SCSI command is sent) and errno is set to EPERM (operation not permitted).
Both the sg driver and the block layer SG_IO code
use internal tables to enforce the permissions shown in the above table (allow_ops and cmd_type [safe_for_read and safe_for_write] respectively). This technique doesn't scale well, since more advanced command sets (e.g. OSD) use service actions (and one
opcode: 0x7f in the case of OSD). There may also be overlap in opcode usage between command sets, for example between SBC, MMC and SSC.
CAP_SYS_RAWIO from a user process
While root processes usually have CAP_SYS_RAWIO, processes running under a user's ID (i.e. non-root) typically don't. Hence non-rootprocesses may not be able to use SG_IO to send SCSI commands that require CAP_SYS_RAWIO. This may occur even if the permission bits of the device node file allow for read or write access, user processes will receive EPERM when using SG_IO.
By default the capability to assign capabilities to other processes (CAP_SETPCAP) is limited to very few processes, such as certain
kernel threads. Changing this default would require to change and recompile the kernel.
Processes which are forked by a root process and call setuid later will lose the CAP_SYS_RAWIO capability the parent root process
(and the child before the setuid) had. However, the child can preserve the capabilities of the root process in the permitted set and raise it after the call of setuid:
/* ... in child after fork(), still running as root ... */
prctl(PR_SET_KEEPCAPS, 1, 0, 0, 0);
setuid(...);
cap_set_proc(cap_from_text("cap_sys_rawio+ep"));
This way a user process with a parent root process can 'get back' the required capabilities to directly send SCSI commands to a
device via SG_IO.
The above technique may be of use to daemons that are started with root permissions (most are) and then changes to another user
after a fork(). It is not obvious to the author how utilities that use the SG_IO ioctl on device nodes that require CAP_SYS_RAWIO for some or all SCSI commands (e.g. nodes associated with the sd and st drivers) can use the above technique.
SG_IO and the st driver
In order to implement its user space API, the st driver has to maintain information about where the read head is with respect tothe structural elements of the tape (filemarks, beginning of tape, end of data).
Because the streaming device SCSI commands don't have addresses, the st driver has to know what commands have been sent. When reading, the filemarks are noticed when a read fails and sense data
is fethed. If SG_IO is mixed with tape commands, the st driver may lose information (it does not look at the SG_IO commands and results). Because of this, the st driver may not implement the semantics the user expects. If the user accepts this or knows when
using SG_IO does not cause information loss, then using SG_IO is OK.
So mixing st driver read, write and ioctl commands with SCSI commands sent via SG_IO that change the state of the tape is not recommended.
This applies whether the SG_IO SCSI commands are sent via st or sg device nodes.
Maximum transfer size per command
The largest amount of datathat can be transferred by a single SCSI command is often a concern. Various SCSI command sets (e.g. SBC-3 for disk READs and WRITEs, SSC-3 for tape READs and WRITEs, and SPC-4 for READ+WRITE BUFFER) allow very large data
transfer sizes but Linux is not so accommodating. The Host Bus Adapter (HBA) could have transfer size limits as could the transport and finally the SCSI device itself. In the latter case SBC-3 defines a "Block Limits" Vital Product Data
(VPD) while SSC has the READ BLOCK LIMITS SCSI command. SBC-3's optional Block Limits VPD page contains both maximum and optimal counts. In the author's opinion that latter distinction is very important:
the block susbsystem should try and use optimal sizes while pass through users should only
be constrained by maximum sizes. Also if a pass through user exceeds a maximum transfer size imposed by a SCSI device, then the device can report an error. There is an underlying assumption that the applications using a pass through interface know what they
are doing, or at least know more than the various kernel susbsystems. On the other hand, the kernel has the responsibility to allocate critical shared resources such as memory.
In the past, Linux used a single, "big-enough", block of memory for the source or destination of large data
transfers. Then scatter-gather lists where added to break transfers up into smaller (often "page" size (4 KB on i386 architecture)) chunks which made memory management easier for the kernel. Now, in the lk 2.6 series, the single block of memory option is being
phased out.
The Linux SCSI subsystem imposes a 128 element limit on scatter gather lists via its SCSI_MAX_PHYS_SEGMENTS define. The way various
memory pools are allocated by the linux SCSI subsystem, SCSI_MAX_PHYS_SEGMENTS could be increased to 256. Associated with each type of HBA there is normally a low level driver (LLD). Each LLD can further limit the maximum number of elements with the scsi_host_template::sg_tablesize
field. Prior to lk 2.6.16 the sg and st drivers used the .sg_tablesize field only,
since lk 2.6.16 those drivers are also constrained by SCSI_MAX_PHYS_SEGMENTS. This leads to a potential halving of the maximum transfer size. Many LLDs set the .sg_tablesize field to SG_ALL (which is 255) but they may as well set that field to 256 unless the
HBA hardware has a constraint.
User space memory may be allocated as the source and/or destination for DMA transfers from the HBA (i.e. direct IO). Even if the
user space allocated a large amount of memory with a single malloc(), the HBA DMA element typically has a different view of memory. This view may well contain many "page" size discontinuous pieces. This has the effect of using up, or perhaps exhausting, scatter-gather
elements.
The sg driver attempts to build scatter gather lists with each element up to SG_SCATTER_SZ bytes large. This define is found in
include/scsi/sg.h and has been set to 32 KB for some years. That is 8 times the page size (of 4 KB) on the i386 architecture. Some users who need really large transfers increase this define (and it is best to keep it a power of 2). However since lk 2.6.16
another limit comes into play: the MAX_SEGMENT_SIZE define which is set to 64 KB. MAX_SEGMENT_SIZE is a default and can be overridden by the LLD calling blk_queue_max_segment_size().
In lk 2.6.16 two further LLD parameters come into play even when the sg (and st) driver is used. These are scsi_host_template::max_sectors
and scsi_host_template::use_clustering .
The .max_sectors setting in the LLD is the maximum number of 512 byte sectors allowed in a single SCSI command's scatter gather
lists (for data
transfers). Yes, that is a strange limit when trying to send a SCSI WRITE BUFFER command to upload firmware. Sysfs makes the LLD's .max_sectors setting visible (converted to kilobytes) in /sys/block/sd<x>/queue/max_hw_sectors_kb . The maximum allowable value
in a LLD's .max_sector seems to be 65535 (0xffff in hexadecimal). This limits the maximum transfer size to (32*1024*1024 - 512) bytes, assuming other limitations have been overcome. [The 65535 sector limit is because Scsi_Host::max_sectors has type "unsigned
short". Hopefully this type is expanded to "int" in the future (or removed).]
The .use_clustering field should be set to ENABLE_CLUSTERING . If not, the block subsystem rebuilds the scatter gather list it gets
from the sg driver with page size (e.g. 4 KB) elements. [Actually is does that anyway, but when ENABLE_CLUSTERING is set, it coalesces them again!]
Conclusion
In some situations, sending commands via the SG_IO ioctl may interfere with a higher level driver's use of a device. Users of theSG_IO ioctl should be aware that they are using a powerful, but low level facility, and write code
accordingly. An example of this would be a utility to perform self tests on a disk: "background" self tests should be preferred over "foreground" self tests if there is a chance the computer may be using a file system on that disk at the time. Even a short
foreground self test may take up to two minutes which is a long time to lock out a file system.
Return to main page.
Last updated: 26th July 2008
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Can not find a java.io.InputStream with the name [] in the invocation stack 解决办法
- nullnullhow to read directory name using std c in the linux
- How to sleep in the Linux kernel?
- QUEUEING IN THE LINUX NETWORK STACK
- Linux Device Drivers 3rd Edition Data Types in the Kernel
- the diary of sleep jobs & fg command line in linux
- struts2下载文件时严重: Can not find a java.io.InputStream with the name [targetFile] in the invocation
- Linux驱动开发④--unknown field ‘ioctl’ specified in initializer .ioctl = xxx
- linux使用mysql 在输入了mysql 之后出现 the programe can be found in the following packages
- sockfs: the most simple file system in linux kernel
- How do I configure the iscsi-initiator in Red Hat Enterprise Linux 5?
- Basic Data Structures and Algorithms in the Linux Kernel
- The commpression command in the linux (gzip , tar)
- Can not find a java.io.InputStream with the name [downloadFile] in the invocation stack.
- The_Linux_Kernel_Module_Programming_Guide_2.6中文版
- linux系统使用sudo命令报xxx is not in the sudoers file.This incident will be reported.的解决方法
- how to read directory name using std c in the linux
- SSH 文件下载发生异常： Can not find a java.io.InputStream with the name [inputStream] in the invocation sta
- Linux下 ntp 时间同步服务ntpd 出现 the NTP socket is in use, exiting 解决
- look the files size in linux system