25 July 1995

Several people have reported problems while attempting to write large ntuple files via HBOOK. The jobs tend to end abruptly with cryptic error messages about some quota being exceeded.

The usual method for writing an ntuple file is illustrated by the following code fragment.

		PARAMETER	(NWPAW=200000)
	c
	c	**************************************************************
	c	* The size of the HBOOOK store in PAW is 2000000 words.  If  *
	c	* you find you need to make it larger than that here, you    *
	c	* may get nasty messages from PAW telling you it is out of   *
	c	* memory. If that happens, you are also out of luck.         *
	c	**************************************************************
	c
		COMMON /PAWC/	HSPACE(NWPAW)   
	     	CHARACTER*8	MCNAM (10)  
	c     
		DATA LUN	/37/
		DATA MCNAM	/'x1','x2','r','a','b','rdiff','r5',
	     +		         'rshift','sqrt','mult'/
		DATA NWORDS	/1000/
	c
		CALL HLIMIT (NWPAW)   
	c    
	C--	Open a unit for hist output.
	c
		call hropen(lhst,'NTUPLE','NTUPLE.DAT','N',nwords,istat)
		if(istat.ne.0)  then
		  print *,' Trouble with HROPEN.  ISTAT = ',istat
		  stop
		end if
	c--
	c      	Book the ntuple
	c
		CALL HBOOKN (110, 'Junk Ntuple', 10, 'NTUPLE', NWORDS, MCNAM)   
				.
				.
			Some loop to generate 10-tuples and
	     		call HFN to enter them into the file
				.
				.

A look at the code for HROPEN revealed two details. HROPEN will accept an option string containing the letter Q which is taken to mean that the user wants to specify his quota of disk records for the ntuple file instead of letting HBOOK choose a default value based, among several other things, on the size of the PAWC array. The number of these disk records is passed through one element of the ZEBRA COMMON block named QUEST. HROPEN will accept a value no greater than 65000 for that quota. Two important points emerge immediately.

• There is a limit to the size of any ntuple file. There is no such thing as an open-ended ntuple file.
• Within certain restrictions, the user can alter but NOT eliminate that limit.

The code fragment above was modified as follows.

	c
		PARAMETER	(NWPAW=200000)
	c
	c	**************************************************************
	c	* The size of the HBOOOK store in PAW is 2000000 words.  If  *
	c	* you find you need to make it larger than that here, you    *
	c	* may get nasty messages from PAW telling you it is out of   *
	c	* memory. If that happens, you are also out of luck.         *
	c	**************************************************************
	c
		COMMON /PAWC/ 	HSPACE(NWPAW)
		COMMON /QUEST/	IQUEST(100)  
		CHARACTER*8	MCNAM (10)  
		DIMENSION	VECTOR(10)
	c
		DATA LUN	/37/
		DATA MCNAM	/'x1','x2','r','a','b','rdiff','r5',
	     +		         'rshift','sqrt','mult'/
		NWORDS		/1000/
	c
		CALL HLIMIT (NWPAW)   
	c    
	C--	Open unit for ntuple output.  At the same time, set the
	c--	maximum size of the file (in records) via IQUEST(10).
	c--	N. B. This is limited to 65000.
	c
		IQUEST(10) = 65000
		call hropen(lhst,'NTUPLE','NTUPLE.DAT','NQ',nwords,istat)
		if(istat.ne.0)  then
		  print *,' Trouble with HROPEN.  ISTAT = ',istat
		  stop
		end if
	c--
	c--	Get some statistics about currently open RZ files.  This is
        c--     really unnecessary razzle-dazzle except for debugging.
	c--
		CALL RZSTAT('//NTUPLE',99,' ')
	c
	c	Book the ntuple, find out how many to generate and get going.
	c
	c	N. B.  The fourth argument to HBOOKN controls whether the
	c	  ntuple is memory resident or disk resident.  If it is
	c	  just ' ', the ntuple is memory resident and there must
	c	  be enough space for it in the HBOOK store.  If it is
	c	  the same character string as the second argument of an
	c	  earlier HROPEN call, then the ntuple is disk resident
	c	  on the file identified by the corresponding LUN.
	c
		CALL HBOOKN (110, 'Junk Ntuple', 10, 'NTUPLE', NWORDS, MCNAM)   
		TYPE 10
	10	FORMAT(/1X,'Enter the number of ntuples to generate: ',$)
		ACCEPT *,N
		DO I = 1,N
			.
     		Generate and fill the ntuples.
			.
		END DO
	c
        c--     The next four statements are again unnecessary razzle-dazzle
        c--     except when doing serious debugging.  The information may be
        c--     useful to keep track of quota usage for BIG files.
        c--
		TYPE 991
	991	FORMAT(/1X,' Ntuple statistics on completion of generation')
		CALL RZLDIR('//','A')
		CALL RZSTAT('//NTUPLE',99,' ')
        c
		CALL HROUT(0, ICYCLE, ' ')
		CALL HREND('NTUPLE')
		STOP
		END    

What should one expect from this setup? Note that we have specified 65000 disk records of 1000 words each for 10-tuples. That means we expect to get 100 ntuples into one disk record so that 65000 records correspond to 6.5 million ntuples if we have accounted for everything properly. When this program was run with 6.5 million ntuples requested, ZEBRA produced an error message about being out of space after about 3.2 million ntuples were generated. Clearly something else was getting in the way. A ZEBRA dump showed that there was almost no action in the user divisions of the dynamic store but that the system division had grown to take up all the available space. Thus, the size of PAWC plays a role in all this as well. A bit of experimentation varying the size of PAWC leads to a very ad hoc and empirical rule of thumb. HBOOK requires 6 words of dynamic store for each direct access record written in addition to a fixed minimum of about 10K words for other storage. All of this is in the system division and must be counted in addition to the storage needed by the user program. Our request for 6.5 million ntuples of 10 words each was intended to fill 65000 records. According to the empirical rule, this would also require a minimum of 6 * 65K + 10K = 400K words of memory for PAWC and we only reserved 200K. Note that the amount of space in PAWC is a multiple of the number of records and not of the number of ntuples.

If we had chosed a record size of 8000 words instead of 1000 words, we could accomodate 8 times as many ntuples for the same size PAWC. This is the reason behind the folk lore about writing "large" records. Note that a record size much greater than 8000 words is strongly discouraged for a number of reasons. Among these are

• Some machines will simply not write records that large.
• Even if machine A will write a large record, machine B may not read it, so that portability is lost.
• The automatic record size recognition code in PAW does not work if the record size is too big.

The price to be paid for doing that is that more space is required in PAWC for the record buffer. It is still a good idea because the buffer is of fixed size and is used repetitively while the other piece grows monotonically at the rate of 6 words per record or, in this case, 6 words per 800 ntuples.

All of this "hidden" overhead is a consequence of the decision to implement ntuple storage in the form of direct access files. This requires HBOOK to build and maintain a mapping between any arbitrary ntuple and its location in the file. Since it assumes the user may wish to recover any arbitrary subset of these ntuples in the same job, that mapping is kept in memory in the dynamic store and space must be reserved for it.

The conclusion from all this is rather straightforward. Given the current design, there is no such thing as an open-ended ntuple file. The user must choose the size of PAWC and set record quotas to accommodate some number of ntuples. If the user is unable or unwilling to limit the size of the ntuple file, HBOOK will.

Security, Privacy, Legal