

@SUBJECT:Re: **HPFS, what is it???**                                 N
From: jahk@uiuc.edu (Jack Tan)
Subject: Re: **HPFS, what is it???**
Date: 24 Jul 1995 16:48:37 GMT
Reply-To: jahk@uiuc.edu


I've written a small article about HPFS.  I hope this helps a little.


     _________________________________________________________________
   
                    THE HIGH PERFORMANCE FILE SYSTEM (HPFS)
     _________________________________________________________________
   
   OS/2 was designed to overcome the limitations of DOS, and one of its
   weapons is the High Performance File System (HPFS). HPFS was designed
   specifically for OS/2 and addresses the problems of FAT (the DOS
   native file system) in today's computing environment. OS/2 can use the
   DOS-style FAT file system as well as HPFS, and OS/2 can use them side
   by side along with other file systems.
   
   HPFS is a method for storing and accessing files on large storage
   media (e.g., hard disks). It was designed from the ground up for huge
   volumes with thousands of files and files of extreme sizes. It is
   optimized for hard disks (or "fixed disks" or "Winchester drives") and
   leverages the size and geometry of hard disks for even better
   performance.
   
HPFS Advantages

  LONG FILENAMES
   
   The most apparent feature of HPFS is its support for long filenames.
   The DOS limitation of naming files and directories is lifted with
   HPFS: files and directories under HPFS can have up to 254 characters,
   including spaces. The names are stored with case retention (so the
   names look "nice" in a directory listing or folder), but are searched
   with case insensitivity (so the case doesn't matter when accessing a
   file, like under FAT).
   
   The directory listing from the DIR command looks different under HPFS,
   as compared to FAT:
   
    [F:\Images\Bitmaps]dir A* B*

    The volume label in drive F is DATA-2.
    The Volume Serial Number is 26BE:6414.
    Directory of F:\Images\Bitmaps

     3-26-95  11:31p     <DIR>           0  .
     3-26-95  11:31p     <DIR>           0  ..
     2-20-95   7:09p    273128        9850  Aliens.BMP
     2-20-95   7:10p     63486        9850  Alpha.BMP
     2-20-95   7:08p       630        9850  Arcade.BMP
     2-20-95   7:11p    262486        9850  Arctic Wolves.BMP
     2-20-95   7:11p    782366        9850  Arizona.BMP
     2-20-95   7:11p     90054        9850  Aspirin.BMP
     2-20-95   7:12p     22240        9850  Asterix.BMP
     2-20-95   4:43p    225114        9850  Bad for Good.BMP
     2-20-95   7:13p    752730        9850  Basilisk.BMP
     2-20-95   7:15p    325406        9850  Bat Sign.BMP
     2-20-95   7:15p    291090        9850  Batman Posters.BMP
     2-20-95   4:46p     17534        9850  Black Marble.BMP
     2-20-95   4:46p      5432        9850  Blue Granite.BMP
     2-20-95   4:49p     35096        9850  Blues.BMP
     2-20-95   7:16p    133798        9850  Bucky Ball.BMP
           17 file(s)    3280590 bytes used
                        36698624 bytes free

    [F:\Images\Bitmaps]_
   
   The filenames are always written on the right side of listing. This is
   done to accommodate files with longer names. File names which contain
   spaces can be specified by using double quotes around the name. For
   example:
   
		      erase "arctic wolves.bmp"
   
   Files with long names cannot be copied directly to a FAT volume from
   the command line, since the filenames are not compatible. To get
   around this restriction, the file should be moved using the Workplace
   Shell (WPS), which will preserve filenames when moving across file
   systems.
   
  EXTENDED ATTRIBUTE HANDLING
   
   HPFS also provides built-in support for extended attributes (EAs), the
   sizes of which are shown directly to the left of the filenames. EAs
   are used by OS/2 to store information which belong to files, but are
   not part the data. Among other things, custom icons and certain WPS
   names are stored in extended attributes. To get an idea of the stored
   names, consider a folder in the WPS that has a backslash or pipe in
   its name. Both of these characters are illegal in the file system, but
   must be stored somewhere. They are stored in the extended attributes
   for that folder. This is also how the WPS preserves filenames when
   moving files and directories to FAT.
   
  RESISTANCE TO FRAGMENTATION
   
   Unlike FAT, which was intended for floppy disks, HPFS was designed for
   large storage devices, such as hard disks, which can store hundreds or
   thousands of megabytes of information. Its design takes into account
   the geometry of the hard disk for faster access, and HPFS works very
   hard to not fragment files on the hard disk. When multiple files are
   created on the hard disk concurrently, HPFS creates them in separate
   areas, so they will not collide with each other as their sizes
   increase. HPFS defragmentation programs are available (see below), but
   they are seldom needed.
   
  CONSTANT CLUSTER SIZE
   
   HPFS provides a constant cluster size of 512 bytes. This is in
   comparison to FAT, whose cluster size grows with the size of the
   partition. For example, every file on a 300MB FAT partition has a size
   in multiples of 8192 bytes, rounded upwards (despite what DIR
   reports); these same files would occupy only multiples of 512 bytes on
   an HPFS partition. This is very important for large volumes (e.g.,
   files on a 1GB FAT partition use 64K multiples) and volumes with many
   small files (like icon files, and small header, object, or resource
   files).
   
  OPTIMIZED FOR EXTREMES
   
   HPFS very easily handles both very large files and very small files --
   from small batch files to huge databases or lengthy multimedia
   productions. HPFS uses balanced B-tree data structures for extremely
   fast access among thousands files. Moreover, the root directory is
   placed at the seek center of the partition for fast access. HPFS is
   also built to handle multiple concurrent I/O requests, as befits the
   multitasking nature of OS/2.
   
DOS, Windows, and HPFS

   In general, DOS and Windows programs run transparently on HPFS.
   Programs can be installed and run under HPFS without modification
   while running under OS/2. Because of their nature, DOS and Windows
   programs can only see files whose names comply with FAT file-naming
   conventions. This essentially means up to 8 characters and an optional
   extention of up to 3 characters, using a period to separate the
   extension. (The file-naming rules for both HPFS and FAT are listed in
   great detail in the OS/2 Master Help Index.)
   
   Other than the long filenames, DOS and Windows programs automatically
   take advantage of the other HPFS features due to the ultimate handling
   by OS/2. DOS and Windows programs which directly access the hard disk
   (regardless of whether the partition is FAT or HPFS) represent a
   threat to the system integrity. Consequently, these programs are
   blocked by OS/2 and cannot be run. Programs in this category include
   low-level disk-recovery programs. Fortunately, HPFS has its own
   disk-recovery programs.
   
  NATIVE DOS
   
   When native DOS is booted, DOS cannot see HPFS volumes because it only
   understands FAT (and, with some help, CD file systems). Files on HPFS
   drives therefore cannot be accessed. There are several drivers
   available which allow DOS programs to access HPFS volumes:

     * HPFSDOS
     * HPFSRead
     * iHPFS
       
Low-Level Help
   
   HPFS has its own disk recovery programs, such as the GammaTech
   Utilities. HPFSTool [PM,32] is a shareware utility to undelete files
   from HPFS partitions. File Phoenix [PM,32], released through the IBM
   Employee Written Software program, can undelete files from both FAT
   and HPFS volumes. Jack Grey has written a defragmentation program for
   HPFS volumes [VIO,32]. Graham Utilities for OS/2 [VIO,16] also has
   several HPFS tools, including applications to mark bad sectors, view
   HPFS extents, undelete files, defragment HPFS drives, view internal
   HPFS information, and show the layout of HPFS.
   
HPFS Files

  HPFS: AN INSTALLABLE FILE SYSTEM
   
   HPFS is an implementation of the Installable File System (IFS) model
   of OS/2. IFSes allow different file systems to coexist and allow files
   to be accessed transparently, regardless of underlying differences in
   structure. The IFS model, inherent in OS/2's design, allows FAT, HPFS,
   and other file systems to exist simultaneously -- various file systems
   in use concurrently under a single operating system.
   
   The IFS model also allows authors to write their own file systems as
   well. A common example of another IFS is CDFS, the CD-ROM file system.
   Other file systems include the Toronto Virtual File System and
   ZipStream file system. IFSes are specified by an IFS=... statement
   in CONFIG.SYS.
   
  HPFS SUPPORT FILES
   
   HPFS support is given by the statement IFS=\OS2\HPFS.IFS... in
   CONFIG.SYS, which looks similar to this:
   
	 IFS=C:\OS2\HPFS.IFS /CACHE:64 /CRECL:4 /AUTOCHECK:C
   
   The various parameters are described in the Command Reference in the
   Information folder; they can be accessed via HELP HPFS from the
   command line. These parameters can be adjusted to give greater
   performance. The default cache settings can be changed by inserting
   
			RUN=\OS2\CACHE.EXE ...
   
   into CONFIG.SYS with the proper parameters. Use HELP CACHE to see the
   documentation for this program.
   
  PREPARING HPFS
   
   To prepare a partition as HPFS, the HPFS.IFS file must be installed as
   above. The FORMAT command is used to prepare the partition (known as
   formatting). Assuming new HPFS partition is partition C:, this will
   format the partition:
   
			  FORMAT C: /FS:HPFS
   
   Under OS/2 Warp, this is a quick format that does not check the hard
   disk for physical errors. To force a long format which does check for
   physical errors, use:
   
			FORMAT C: /FS:HPFS /L
   
   This is the default behavior for OS/2 2.x.
   
   Note that all partition contents will be destroyed when the hard disk
   is formatted, so any contents on the partition must be backed up or
   otherwise saved before the formatting takes place. The PartitionMagic
   product from PowerQuest will convert FAT partitions to HPFS without
   this destructive formatting.
   
  FILES FOR CHKDSK
   
   The CHKDSK program is used to find and correct errors in HPFS (as well
   as other media). CHKDSK also updates internal HPFS structures, such as
   the pool of hotfixes. CHKDSK under OS/2 is much more powerful than its
   DOS counterpart.
   
   If there are open files (i.e., files in use) on an HPFS partition,
   CHKDSK will not accept the /F parameter. Such partitions include the
   OS/2 boot partition and the partition on which the swapfile resides
   (given by the SWAPPATH=... line in CONFIG.SYS). To get around this,
   the system must be booted from a different source.
   
   Many people use the BOOTOS2 utility to create an emergency boot
   diskette. Otherwise, the OS/2 Installation Diskette and Diskette 1 can
   be used to boot to a command line: when the boot process finishes,
   pressing F3 (for OS/2 Warp) or Esc (for OS/2 2.x) will bring up a
   command line. CHKDSK should be run from Diskette 2.
   
   In order for CHKDSK to work, the directory which contains UHPFS.DLL
   must be specified in the LIBPATH in CONFIG.SYS. For OS/2 2.x, the
   directory must be in CONFIG.SYS when the system is booted. For OS/2
   Warp, the user can set the environmental variable ENDLIBPATH to
   dynamically append directories to LIBPATH. Similarly, BEGINLIBPATH
   prepends directories to LIBPATH. For example:
   
		      SET ENDLIBPATH=A:\OS2\DLL;
   
   If UHPFS.DLL is not found, the output will be very short, and the
   drive will not be checked.
   
    [A:\]CHKDSK F: /F
    The current hard disk drive is: F:
    The type of file system for the disk is HPFS.
    The HPFS file system program has been started.

    [A:\]_
   
   If UHPFS.DLL is found, the output should be similar to this:
   
    [A:\]CHKDSK F: /F
    The current hard disk drive is: F:
    The type of file system for the disk is HPFS.
    The HPFS file system program has been started.
    CHKDSK is searching for lost data.
    CHKDSK has searched 100% of the disk.
        207004 kilobytes total disk space.
           336 kilobytes are in 280 directories.
        171402 kilobytes are in 4101 user files.
          1581 kilobytes are in extended attributes.
          2048 kilobytes are reserved for system use.
         31636 kilobytes are available for use.

    [A:\]_
   
   Note that the values are in kilobytes rather than bytes, as CHKDSK on
   FAT would show. In order for CHKDSK to produce error messages, the
   file OSO001.MSG ("oh-ess-oh-zero-zero-one") must be in the DPATH.
   OSO001.MSG normally resides in \OS2\SYSTEM\ in an OS/2 installion. To
   view the current DPATH, just type DPATH at the command prompt. To
   append C:\OS2\SYSTEM to the DPATH, use the command
   
		   SET DPATH=%DPATH%;C:\OS2\SYSTEM;
   
Requirements
   
   While any installation of OS/2 beyond version 1.1 can use HPFS, it is
   not recommended for everyone. HPFS uses more overhead on the disk than
   FAT for bookkeeping, so the savings in disk space are usually not
   apparent until the partition size is greater than 60-120MB (the amount
   varies depending on who you ask). Those with hard disks with many
   small or very large files will benefit more than others. HPFS also
   requires an IFS driver and has its own cache, both of which increase
   its memory overhead. Consequently, 8MB or more RAM is generally
   recommended for those who wish to use HPFS.
   
HPFS Values
   
   As mentioned above, the size of an allocation unit in HPFS is always
   512 bytes, regardless of the partition size. Under OS/2 2.x and OS/2
   Warp, the maximum partition size is 512GB, with a maximum file size of
   2GB.
   
More Information
   
   Online information on HPFS can be found in the Command Reference by
   typing HELP HPFS.
   
   More detailed information on HPFS can be found in the September 1989
   Microsoft Systems Journal article entitled "Design Goals and
   Implementation of the New High Performance File System," by Roy
   Duncan. The article is given in normal text, PostScript (viewable with
   Ghostview for OS/2), and LaTeX (viewable using dvipm from the emTeX
   package). A version in Microsoft Word 2.x format, including the
   sidebar article and figures, is also available.
   
   The 27 June 1995 issue of PC Magazine also has an excellent article on
   HPFS by Brian Proffit. The article, "OS/2's High Performance File
   System," is in the "Tech Notes" section of the magazine.

     _________________________________________________________________
   
    Jack Tan / jahk@uiuc.edu
     _________________________________________________________________

-- 

Jack Tan
jahk@uiuc.edu
[TEAM OS/2]

