| |
|
|
|
| |
|
| Variable |
|
| In programming, a named storage location capable of containing a certain type of data that can be modified during program execution. |
|
| |
|
| Virtual Memory (or Virtual Memory Addressing) |
|
Program code and data are stored in random access memory (RAM) chips - also known as physical memory. The remaining three types of memory are: registers in the CPU, CPU caches (generally some kind of static RAM) both inside (L1 cache) and adjacent to the CPU (L2 cache), and main memory (generally dynamic RAM) which the CPU can read and write to directly and reasonably quickly. Compared to the relatively slow hard disk drive, this kind of memory is extremely fast, but a limited resource. So in addition to physical memory, Windows implements, using the Windows Executive’s Memory Manager, a virtual memory system based on a flat (linear) or memory management scheme, address space, reserved on a portion of the hard disk drive, that provides each process with the illusion of having its own large, private address space, also known as page pool, swap- or page file. Virtual memory addressing can then be utilised to exceed available physical memory to the hard disk drive if and when necessary, and marks the associated physical memory as being available for use. Moreover, virtual memory addressing is typically used in pages memory systems which in turn with memory swapping (or anonymous memory paging) memory pages stored in primary storage are written to secondary storage (called a swap-, or swap partition), freeing faster primary storage for immediate use. In paging, the low order bits of the binary representation of the virtual address are preserved, and used directly as the low order bits of the actual physical address; the high order bits are treated as a key to one or more address translation tables – the mapping from virtual memory to physical page numbers, which provide the high order bits of the actual physical address. Specifically, the operating system stores the address translation bale in a data structure called a page table. Note: Windows NT family operating systems use paged memory and virtual memory addressing, even if virtual memory is disabled during run time. At run time, the Memory Manager, with the assistance of hardware, translates or maps, the virtual addresses (virtual memory addressing) into physical addresses, where data is actually stored. By controlling the protection and mapping, the operating system can ensure that individual processes do not bump into one another or overwrite operating system data. Note: Virtual memory provides a logical view of memory that may not correspond to its physical layout. The Memory Manager also provides the underlying support for the Cache Manager. This transfer occurs in 4K chunks (but the page size is typically in the range of 512 through to 8193 bytes) memory, called pages; thus the process is called paging out. When an application requires program code or data residing in memory that has been paged out, Windows pages out other portions of memory used less recently, and swaps the required pages into physical RAM - paging in. This swapping process is much slower than using RAM, but it is usually preferable to allowing a request for memory to fail.
Because most systems have much less physical memory than the total virtual memory in use by running processes, the Memory Manager transfers, pages some of the memory contents to the hard disk drive. Paging data to the hard disk drive frees physical memory so that it can be used for other processes or for an operating system itself. When a thread accesses virtual addresses that has been paged to the hard disk drive, the Virtual Memory Manager loads the information back into memory from the hard disk drive. Applications do not have to be altered in any way to take advantage of paging because hardware support enables the Memory Manger to page without the knowledge or assistance of processes or threads.
Windows attempts to keep a certain amount of physical memory available so that when an application makes a request for memory, that request can be fulfilled immediately. At a convenient time, that is, when the system is otherwise idle, Windows will copy the least recently used pages of memory to the hard disk drive, even before an application requests more memory than is available. Since the memory has already been backed up to disk, Windows does not have to page out but can simply mark the physical memory as being free immediately and can allocate the memory to the requesting application. In addition, Windows tries to keep the most frequently used pages of memory in physical RAM, rather than allowing them to be paged out. The amount of physical memory available for the operating system and for applications is called the process working set, or simply the working set. Since paging is relatively slow, the system generally performs better with the largest possible working set. However, Windows also requires a considerable amount of memory for the file system cache.
The file system cache is intended to make disk access more efficient. In order to do this, the caching system uses physical memory - which again is very fast, compared to a hard disk drive - to act as a holding area between working memory and the hard disk drive. When Windows or an application reads data from the disk, the cache will try to anticipate which data on the hard disk drive is likely to be used next. The caching system will retrieve that data in the background, and store it in memory, in the hope that the next disk request can be supplied from relatively fast memory, rather than from the relatively slow hard disk drive. When Windows or an application writes data to the disk, the caching system will hold some of the most recently used data in the cache, again in the hope that the same data will be needed again soon. In addition, the data will be copied to the disk as soon as possible, but the system may wait a few moments until an appropriate time. Caching systems usually make overall performance much faster, at the expense of reducing the amount of memory available for Windows, the applications, and the data being generated by them.
The size of the virtual address space varies for each hardware platform. On 32-bit x86 systems, the total virtual address space has a theoretical maximum of 4GB or 2^32 bites. By default, Windows allocates half the address space (the lower half of the 4GB virtual address space, from (0x00000000 through to 0x7FFFFFFF) to processes for their unique private storage and uses the remaining half (the upper half, 0x80000000 through to 0xFFFFFFFF) for its own protected operating system memory utilisation. The mappings of the lower half change to reflect the virtual address space of the currently executing processes, but the mapping of the upper half always consists of the operating system’s virtual memory. It is possible for Windows XP SP2 onwards to set the boot-time option using the Boot.ini switch /3GB and /USERVA to give processes running specially marked programs the ability to use up to 3GB of private address space; leaving the operating system with 1GB of private address space instead of 2GB. This option would only be applicable if running a database server for example.
On 64-bit x86 systems, the total virtual address space is theoretically much larger: 7152GB and 8192GB on Itanium and x64 systems respectively.
Note: If a page that is marked as unavailable (perhaps because it is not present in physical memory, but instead is in the swap area), when the CPU tries to reference a memory location in that page, the Memory Manager Unit (MMU) responds by raising an exception (commonly called a page fault) with the CPU, which then jumps to a routine in the operating system. If the page is in the swap area, this routine invokes an operation called a page swap, to bring in the required page. The MMU is a class of computer hardware components responsible for handling memory accesses requested by the CPU. Among the functions of such devices are the translations of virtual addresses to physical addresses (i.e., virtual memory management), memory protection, cache control, and bus arbitration. |
|
| |
|
| Volume |
|
An area of storage on a hard disk drive that functions as though it were a physically separate disk - by a single drive specifier. A volume is created when a partition is formatted by a filesystem, e.g., FAT or NTFS, and has a drive letter assigned to it. A single hard disk drive can have multiple basic volumes or dynamic volumes that span across multiple hard disk drives. For example, in My Computer and Windows Explorer, volumes appear as local disks.
See: Basic Volume (Simple Volume), Dynamic Volume, and Spanned Volume. |
|
| |
|
| Volume Boot Records (VBR) |
|
Whereas the Master Boot Record (MBR) is located at cylinder 0, head 0, and sector 1 (absolute 0 or CHS 0, 0, 1) of the bootable hard disk drive, the boot sector resides on each volume including the bootable hard disk drive. The Volume Boot Record (also known as the Volume Boot Sector or Partition Boot Sector) is the first sector on any area of the drive addressed as a partition or volume (or logical dos drive), floppy diskette or similar storage media device that contains code for bootstrapping programs (usually, but not exclusively operating systems) and stored in other regions of the volume. On a floppy diskette or removable cartridge, for example, this sector is the first one on the hard disk drive because DOS recognises the hard disk drive as a volume without the need for partitioning. On a hard disk drive, the VBR or sectors is located as the first sector within any hard disk drive area allocated as a non-extended partition, or any area recognised as a logical volume. Refer to the table showing the format of the MBR and its partition tables showing the phyical relationship between this VBR and the other data structures on a hard disk drive.
Suffice it to say, either the System ROM BIOS for the floppy diskettes boot record or hard disk drive’s MBR (or boot manager) loads the VBR of the active partition on the hard disk drive, indirectly and directly respectively. The program code in the VBR is given control of the system; it performs some tests and then attempts to load the first system file - for DOS/Windows 9x & Millennium the file is IO.SYS). The VBR, similar to the MBR, is transparent to the running system; it is outside the data area of the hard disk drive on which files are stored. On the FAT partition, VBR is created using the FORMAT command (high-level format) included with DOS, Windows 9x & Millennium. Hard disk drives on FAT systems have a VBR at the beginning of every logical dos drive area allocated on the hard disk drive, in both primary and extended partitions. Although all the logical drives contain the program area as well as a data area, only the program code from the VBR in the active partition on the hard disk drive is executed. The operating system files read the others during boot-up to obtain their data tables and determine the volume parameters. The VBR contains program code and data. The single data table in this sector is called the media parameter block (MPB) or hard disk drive parameter block (DPB). The operating system needs the information this table contains to verify the capacity of the hard disk drive volume as well as the location of important structures such as the FAT. The format of this data is very specific.
Some dual boot systems, such as Ntldr.exe, take copies of the bootstrap code that individual operating systems install into a single partition’s VBR and store them in disc files, loading the relevant VBR content from file after the boot loader has asked the user which operating system to bootstrap.
In certain filesystem formats, in addition to bootstrap code, the VBR contains a BIOS parameter block (BPB) that specifies the location and layout of the principal on-disc data structures for the filesystem.
The VBR resembles the MBR in that it constains two elements:
- Hard disk drive parameter block: the hard disk drive BPB contains specified information about the location and layout of the principal on-disc data structures for the filesystems, e.g., about the volume, its size, the number of hard disk drive sectors it uses, the size of the clusters, and the volume label name.
- Volume boot code: the program or volume boot code is either directly invoked by the machine’s System ROM BIOS or indirectly by the MBR or boot manager (called chain loading). This begins the process of loading the operating system. For DOS and Windows 9x & Millennium filesystems, this is the IO.SYS file.
See: BIOS (Basic Input/Output System), BIOS Parameter Block (BPB), Boot Code (executable instructions within the MBR)
Boot Sector, Bootstrap, Bootstrap Loader, CHS (Cylinder, Head, Sector), Extended Partition Boot Record (EPBR), FAT (12-, 16-, and 32-bit), File Allocation Table, Filesystem, IO.SYS, LBA (Logical Block Addressing), Master Boot Record (MBR), NTFS (New Technology Filesystem), Partition, Primary Partition, Partition Table, and Volume Boot Records (VBR). |
|
| |
|
| Volume File |
|
A system file NTFS metadata file (filename: $Volume) that contains the volume name and version of NTFS for which the volume was formatted. The volume version number is 3.0 and 3.1 for Windows 2000, and Windows XP and Windows Server 2003 respectively. A bit that when set, signifies that a hard disk drive corruption has occurred and must be repaired by the Check Disk utility.
See: Check Disk (chkdsk.exe), NTFS (New Technology Filesystem), and Volume. |
|
| |
|
| Volume Label |
|
| An identifier or name of up to 11 characters that names a disk. |
|
| |
|
| Volume Mount Points |
|
System objects in the version of NTFS included with Windows NT family operating systems (excluding Windows NT V4.0) that represent storage volumes in a persistent, robust manner. Volume mount points allow the operating system to graft the root of a volume onto a directory.
See: Directory, Mount Points, NTFS (New Technology Filesystem), and Volume. |
|
| |
|
| Volume Shadow Copy (NTFS filesystem for Windows XP Professional & Server 2003 only) |
|
The Volume Shadow Copy Service (VSS) (\System32\Vssvc.exe), a volume shadow copy or snapshot (point-in-time) engine, provides the backup infrastructure implemented for Windows XP Professional and Server 2003 to carry out a complex yet totally reliable service. Moreover, VSS also offers more complicated scenarios that until now were not conceivable under Windows via a filter driver called volsnap.sys beneath the file system was available. VSS technology, increasingly popular, is on by default for both operating systems - a mechanism to provide a framework for creating a co-ordinated and consistent point-in-time copy of hard disk drive volumes also known as shadow copies. Microsoft supplies this functionality of the provider so that vendors do not have the inconvenience to write such a service themselves. Microsoft also provides some other components also, but their functionality is limited either in terms of working with a specific application or in terms of the range of features provided; hence vendors are expected and encouraged to write these components.
There are several possible ways in which a volume snapshot or a shadow copy is created. The only real difference is the approach in which the write operations are duplicated.
The VSS framework or architecture consists of four types of modules that co-ordinates VSS compliant backup applications (requestors), business applications (writers), volumes shadow copy service and storage snapshot technology (providers) to enable reliable storage backup and recovery.
A provider is typically implemented as a finite state machine. The state machine transitions forms one state to another upon receipt of an event that is generated by the shadow copy service. A provider will receive an event (generated by the shadow copy service), but exactly which event depends on the current state of the provider and whether an error has occurred. In other words, a provider has a preferred event that it expects to arrive, allowing it to transition to the next normally expected state. However, an error may occur and the provider may receive a different (from the expected) event, and the provider should be able to handle that as well.
The Microsoft shadow copy framework in Windows XP Professional and Windows Server 2003 provides some core functionality needed for storage management, including the following:
- Defining the point in time in which the snapshot is triggered.
- Providing synchronisation services among applications, databases, operating systems, and file systems to flush cached data, suspend writers, create the snapshot, and provide notification to applications, databases, and operating system elements so that they can resume normal mode of operation.
The Microsoft shadow copy framework can handle a set of volumes for which a snapshot must be created as a single set. The only behaviour supported is “all or nothing”; that is, if any one operation fails, all of them fail.
The Microsoft shadow copy framework also issues a request (to the snapshot provider) to delete the snapshot when the requestor application is carried out with the snapshot. If the snapshot needs to be available later, the snapshot provider or requestor application must provide the necessary functionality. Independent Software Vendors (ISVs) can build applications that can create, catalogue, and manage multiple snapshots using the shadow copy architecture; however, such applications are not part of the Windows Server 2003 product available from Microsoft.
Snapshot writers are simply the applications that write data (applications simply reading data are not much of a problem). Examples of snapshot writers include Microsoft Exchange, Microsoft SQL Server 2000, SAP, and Oracle. Microsoft and third-party ISVs are both expected to develop snapshot writer–aware applications. In particular, snapshot writers will receive two events from the snapshot service indicating that they should freeze all write activity, and a different thaw event indicating that write activity can now happen (hopefully signifying successful creation of the snapshot). Because the applications can define the consistency of data they expect to achieve, they should be able to perform the operation fairly quickly.
Further, it is worthwhile understanding one big advantage of the volume shadow copy service architecture over traditional hardware-based snapshot mechanisms used with Windows 2000 and earlier Windows versions. With these classic versions of creating a snapshot, the hardware based mechanism had no means of determining the state of the application and of the operating system software and cache in particular. This meant that an appreciable percentage of the snapshots were inconsistent. Further, the only way of determining the health of the snapshot created required running an application data consistency checker, which could run for hours.
By comparison, the VSS–based architecture not only attempts to flush caches and hold writes, but it can also determine in a matter of a minute or so, whether the created snapshot is consistent or not. Once the thaw event has been signalled, the volume shadow copy service simply interrogates the writers as to whether they successfully managed to hold their write operations between the freeze and thaw events. If any of the writers failed, the snapshot is deemed to have failed.
Snapshot writers are also expected to provide data required for backup and restore—for example, what files need to be copied, what files need to be excluded, what collection of objects needs to be treated collectively as a single set. This data is stored by the snapshot service in a writer metadata document that is in XML format. Writers can also use this document to store data that is of interest to them. To restore data, the application simply hands the collection of data to the writer application to accomplish the restore operation.
Microsoft has announced that it will ship a SQL Server 2000 and Exchange writer, as well as writers for various other components of the Windows Server operating system. Microsoft is cooperating with ISVs to develop writers for other applications, including Active Directory.
Requestors are typically backup applications that request the creation of a snapshot by making the appropriate API (Application Programming Interfaces) call to the volume shadow copy service provided by Microsoft. What’s interesting here is the fact that the model remarkably simplifies some issues for the backup application writer. The backup application no longer has to solve the difficult issue of where the data to be backed up is, or what part of the data consists of application log files and what special treatment those files merit.
The appropriate writer (e.g., Microsoft SQL Server) is responsible for specifying the files and directories that should be included in the backup. Restore operations also become a lot simpler because the restore application again does not have to locate the data and figure out what files to pass to what API of the application (such as Exchange or SQL). The restore application simply hands the collection of data to the writer (application) and lets it accomplish the restore operation.
The Windows NT family operating system VSS service, written by Microsoft, coordinates the activities for all the snapshot components. In particular, it provides the following:
- A single interface for backup applications or snapshot requestors to deal with, rather than multiple APIs from multiple applications.
- A single interface to produce, manipulate, and delete a crash consistent volume snapshot or ghost volume.
- A single interface to allow different applications and snapshot providers to register and deregister as snapshot writers or snapshot providers.
- Synchronisation and coordination among the various components to accomplish creation, deletion, or movement of snapshots, including backup and restore operations. Hardware providers have the highest priority, software providers are next, and the default Microsoft provided snapshot provider has the lowest priority.
ISVs need not be concerned about writing a VSS service with Microsoft services at hand in various degrees.
Snapshot providers are expected to be written by ISVs and Independent Hardware Vendors (IHVs) to create, delete, and manipulate snapshots. As described earlier in this chapter, snapshot providers need to be written as a COM out-of-proc provider.
The provider may also have a kernel mode component—for example, a filter driver that is located between the file system and the Logical Disk Manager (LDM). This kernel mode functionality may also be optionally implemented in hardware instead. Note: Even a hardware-based provider will still leverage the rest of the functionality provided by the framework—for example, definition of the point in time, I/O synchronisation, and platform for building storage management applications, including backup, restore, and snapshot management applications.
A prime example of a snapshot provider is the volsnap.sys driver that ships with Windows XP Professional and is also expected to ship with Windows Server 2003. This provider uses copy-on-write technology to create the necessary minimal data in a side store to be able to re-create the volume at a given point in time. The assumption is that the amount of free hard disk drive space is indeed available. This provider can handle NTFS, FAT32, and raw volumes on Windows Server 2003. However, this snapshot provider can provide only read-only snapshots and handle only one snapshot per volume. This limitation is in the provider itself and not the infrastructure. ISVs and IHVs can build richer functionality in their providers and writers if they believe it is worthwhile.
Snapshot providers must provide or exceed certain mandatory functionality.
The mandatory functionality is as follows:
- Providers are responsible for locating the storage needed to create the snapshot. The framework provided by Microsoft does not provide any such functionality.
- The provider must mount the snapshot in a different namespace and not have the snapshot mounted as a separate volume. An inspection of Windows XP Professional shows that the Microsoft snapshot provider mounts the snapshot at \Device\HarddiskSnapshotX.
Requestors are typically backup applications that request the creation of a snapshot by making the appropriate API call to the volume shadow copy service provided by Microsoft. From the point of the application writer, the backup application has been simplified so that it is no longer necessary to solve the difficult issue of where the data to be backed up is, or what part of the data consists of application log files and what special treatment those files merit. Moreover, the restore application simply hands the collection of data to the application writer and lets it accomplish the restore operation.
A shadow-copy aware application can receive freeze and thaw notifications to ensure that backup copies of data files are internally consistent.
The VSS is invoked by the requestor and coordinates with it, which is typically a backup application, to create a shadow copy volume, an instant copy of the original volume which must be formatted with NTFS, at the time the backup is initiated. Data is then backed up from the shadow copy instead of from the original volume. The original volume continues uninterrupted to change as the process continues, but the shadow copy of the volume remains constant. This is helpful if users need access to files while a backup is taking place. The source data and the shadow copy volume are independent. The original volume continues to take application changes, while the shadow copy volume remains an exact read-only copy of the original data at the time of the break. Once the backup is completed and the data is saved on the backup media, the shadow copy is deleted. Several features in the Windows Server 2003 operating system use VSS, including Shadow Copies for Shared Folders and Backup.
The VSS framework is designed to manage data on machines where both the application and its related files reside. File-only servers are not supported by VSS because the applications that manipulate the data on these servers are running on other machines and cannot be integrated with VSS on the file server. Often, enterprises have applications that run on workstations separate from where the files are stored. These client applications are not supported by VSS either. However, for non-compliant and legacy writer applications Genie-soft’s File Access Manager (FAM) has added functionally to support legacy writer applications within the VSS framework.
VSS does not provide transactional integrity for most local applications or any remote applications, but instead provides a snapshot having crash consistency (if the computer has no applications that are enabled for the VSS, the data in a shadow copy is considered to be in a “crash consistent” state. All files that were open will still exist, but are not guaranteed to be free of incomplete I/O (read and write) operations or data corruption). While the crash-consistent state does not fully deal with all the issues associated with defining a stable backup set, it has several advantages over the backup set that conventional backup operations would have to use.
Windows Server 2003 also takes advantage of VSS to provide a feature for Shared Folders that lets end users access backup versions so that they can recover old versions of files and folders that may have been deleted or changed. Client systems that wish to take advantage of VSS for Shared Folders must install an Explorer extension called Previous Version Client (\System32\Clients\Twclient). When a client Windows system that has a VSS Explorer extension installed maps a shared form of a folder on a volume for which snapshots exist, a tab named Previous Version appears on the Properties dialogue box for folders and files on the share. The Previous Version tab shows a list of snapshots that exist on the server, allowing the end user to view or copy a file or folder’s data as it existed in a previous snapshot.
The shadow copy of the volume is created at a single point in time and is synchronised across the whole volume (GMB) set. In order to avoid inconsistencies, shadow copies are not taken file by file. FAM does provide transactional integrity for all applications. Depending on this functionality of FAM, to insure the integrity of the backup saved set, configure Genie Backup Manager Professional and Server not to use VSS. When FAM is enabled on these systems, it replaces the default VSS shadow copy provider. Pausing or stopping FAM causes the default shadow copy provider to be re-enabled. When FAM is operating as the VSS shadow copy provider, it provides the same functionality of the default provider and additionally supports VSS backup of FAT volumes. When FAM is enabled it has no interaction with the VSS shadow copy provider. It is possible to enable both FAM and VSS on these systems but his should be avoided as it could lead to excessive consumption of hard disk drive space and memory while both systems provide volume snapshots. Note: GBM Professional and FAM are required; GBM Server has FAM integrated within it.
Note: FAM uses the complicated above the file system approach. A file system filter driver sits above the file system driver for FAT or NTFS and duplicates each IRP (or I/O request packet) sent to the file system driver. This process is rather complex and cannot easily take advantage of snapshot technology provided in hardware.
Note: Writers and providers need to implement a Component Object Model (COM) out-of-proc. COM is an extremely important architecture in the Windows world that allows software to be constructed as a series of objects that interact with each other. These COM objects can be written in a variety of programming languages, such as C, C++, and Visual Basic, to name just a few. COM allows two interacting objects to be locally situated on the same machine or to be running on two separate machines altogether. Either these two interacting objects can both be running in the same process (if they are both on the same local machine), or they may be running in two separate processes. In the case of separate processes, COM provides mechanisms for inter-process communication. To summarize, an out-of-proc provider is simply a provider, implemented as a COM object that runs in a separate process context. Backup applications and business applications must be VSS compliant in order to benefit from the VSS framework. This means they must be integrated with the VSS framework and all elements of the framework (the Requestors, Writers, Providers, and the data being protected) must reside on the same computer system.
High-end storage, which can track metadata on a per-sector basis, allows backup/restore operations to become efficient by allowing changes to be tracked very closely, having backup operations deal with just changed data. To be certain that a snapshot is consistent from an application’s point of view, it is imperative that the operating system – filesystem included, and the application participate in the process of flushing the cache and temporarily suspending write operations while the snapshot is made. The VSS copy service shipping with Windows XP Professional and Windows Server 2003 provides the needed operating system and filesystem support, as well as architecture for the needed application support. Only time will tell how long it takes for vendors adopt and support the volume shadow copy architecture.
See: Open File. |
|
| |
|
| |
|
| |
|
|
