Bug in ext:read-vector

[jcunningham]

I have been trying to read large binary files of floating point data using CMUCL (19c). I thought I would have to do it using some form of FFI and went to comp.lang.lisp for help getting that working. I succeeded. But Duane Rettig at Allegro suggested it would be easier to use 'read-vector. So I tried that as follows:

(let ((vec (make-array 10 :element-type 'double-float)))
  (with-open-file (os "d10.bin")
    (read-vector vec os)
    (print vec)))

where "d10.bin" is a double-float binary file containing 10 elements. When I try to read the file it produces the following error:

Type-error in KERNEL::OBJECT-NOT-DOUBLE-FLOAT-ERROR-HANDLER:

#\Null is not of type DOUBLE-FLOAT

Here is the C-code code I used to produce the file:

#include <iostream>
#include <fstream>
#include <complex>
using namespace std;
int main()
{
  int n=10;
  double *d = new double[n];
  for (int i=0; i<n; i++) 
    d[i] = i;
  FILE *of = fopen("d10.bin", "wb");
  fwrite(f,8,n,of);
  fclose(of);
  return 0;
}

Here is Duane's sample code that he says works in Allegro along with comments: (copied from <​http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/67876101085aee82/05a20cfcd11f8fbe?hl=en#05a20cfcd11f8fbe>) ................................................................... "You must be using a simple-streams implementation from another lisp. Allegro CL doesn't have a KERNEL package.

What you're seeing above is a bug; you should report it to that lisp's support team.

It works fine in Allegro CL (for which you can download the Express Edition for free):

[edited formating for readability on Trac - jkc]

(defvar *x*
  (make-array 10 :element-type 'double-float
     :initial-contents
     (loop for i from 0.0d0 to 9.0d0 collect i))) 

#(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)

(with-open-file (s "z.dat" :direction :io
  :if-exists :overwrite
  :if-does-not-exist :create)
(write-vector *x* s))
80
(defvar *y* (make-array 10 :element-type 'double-float :initial-element 10.0d0))

(with-open-file (s "z.dat")
(read-vector *y* s))

*y*
#(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)

-- Duane Rettig"

Further correspondence with Raymond Toy corroborates this is a bug:

"Yes, this does appear to be a bug in the implementation of read-vector.

You can, however, achieve what you want by opening the file with an element-type of, say, (unsigned-byte 8), instead of the default 'character.

I'll have to read some more to understand how read-vector interacts with the stream element type. It seems, though, that the element-type of the vector overrides the element-type of the stream, more or less. Currently, a stream element type of character basically causes the code to read in characters.

Ray"

If you need more information, let me know and I'll see what I can do.

-jeff

[Raymond Toy]

First I added a bit of markup so that the code bits are formatted better.

Now for the bug report, there are a couple of issues here. CMUCL has an extension, READ-VECTOR, which is similar to the SIMPLE-STREAM's READ-VECTOR, but not quite. In particular, it produces the bug that you see. However, if you (require 'simple-streams), READ-VECTOR will be replaced with the simple-stream version. Then, the following snip of code works as Duane mentions:

(let ((vec (make-array 10 :element-type 'double-float)))
	   (with-open-file (os "d10.bin" :class 'stream:file-simple-stream)
	     (stream:read-vector vec os) (print vec)))

The important part is that you open the file with the appropriate class.

Not sure what to do. We can change READ-VECTOR to match simple-stream READ-VECTOR. (BTW, READ-VECTOR is much faster than STREAM:READ-VECTOR.)

[Raymond Toy]

Change summary to include package qualifier.

This should be fixed in the Sep snapshot. A simple change to *read-simple-array-recognized-types*, adding BASE-CHAR and CHARACTER fixes this.

Closing the report.