SeisSpace and Parallel File Systems: What are You Using?

Here’s a few questions for all you SeisSpace^® doodlebuggers out there. What parallel file system are you using for your secondary storage requirements? Have you ever thought about using a different file system? Have you ever wondered if you could combine both your primary and secondary storage requirements on the same parallel file system — and if you did, what would happen?

Well, stop wondering. Over the last several months, Cray invested the expertise of its oil & gas performance engineering team in testing those questions, along with a few others that you may be interested in. Cray’s team, along with Dan Grygier, CTO of Taming Traces Consulting, used Cray’s CS400™ cluster supercomputer (which we qualified for SeisSpace last year) and three separate storage platforms to look at how workflow performance would be affected if you did a few things:

• Combined primary and secondary data on the same parallel file system
• Used GPFS instead of Lustre^®
• Modified your file system block size
• Used Lustre pools for primary and secondary separation

We ran hundreds of production-level read, write and read-then-write SeisSpace jobs with both GPFS and Lustre, varying the input for block size, and got some very interesting results. (Scroll to the bottom to register for an upcoming free webinar in which we’ll discuss this project.)

Below is a graph of our results that uses the average of the runs for a certain job and file system.

Let me help explain what you see above. The Y axis is in MB/sec (16 GB/sec max). Along the X axis are the SeisSpace jobs we ran in groups while varying joblets, extents, contention and read/write file size. We tested the same workflows for Lustre and GPFS on three separate storage platforms: The Cray^® Sonexion^® platform, IBM’s GL6 and DDN’s GS14KX. We also built both primary and secondary on the same parallel file system for all runs. The areas marked A, B, C and D will be the point of our discussion.

The graph has three lines: the blue line is Lustre on the Sonexion system, the gray is GPFS on the DDN system and gold is GPFS running on IBM. Each point on the graph represents an accumulated average of all the jobs we ran for each workflow. Each workflow run was extremely consistent in its results, so using the averages seemed to be a better way to show the critical differences.

Looking at the graph as a whole, you would think that GPFS is the clear performance winner in our tests. But let’s break this down. The first part of the graph, where areas A, B and C reside, is the area where the majority of SeisSpace workflows would be run in a normal sequence of processing steps. In these areas, I/O is categorized as random. Area D is different. These workflows read and write all of the data within an extent at one time or streaming I/O, very similar to a migration. Now let’s look at the breakdown.

Area A is where 32 joblets write using one to 32 extents. In this area, clearly GPFS functions better with lower extents and evens out with Lustre when extents and joblets become equal. In most production cases, joblets and extents are not this distinctly separated (but I’m told you can’t control SeisSpace users and their input, so this does happen).

Area B is an area where we saw some minor read only and write only issues that we have yet to clarify.

In area C, we have an interesting difference between the systems as a result of a 784 GB write, a read, and then a read-write with load; this area clearly shows GPFS performing better.

For area D, we are testing with both JavaSeis and ProMAX data formats. In the series of troughs and peaks, we are writing and then reading a 6 GB JavaSeis file and then a ProMAX file. We then double the size of the file to 12 GB and follow the same sequence of writes and reads. The workflows all use 32 joblets with varying trace frames. You can see GPFS handles reads with higher loads better in these cases than Lustre. Writes (troughs) are all essentially the same.

In summary:

1. Under the I/O load we produced, running multiple jobs with other non-SeisSpace contention on the system, we found that placing both primary and secondary file systems on the same parallel file system produced little reduction in performance. This will vary as the secondary user load increases causing a possible delay in primary and in the use of the SeisSpace Navigator.
2. Both Lustre and GPFS perform equally well under production workflows and the load we provided. GPFS does have the tendency to provide better performance on edge case requirements and higher read loads.
3. The newest versions of Lustre and GPFS work extremely well right out of the box. The most critical parameter to adjust is block size. We found 4M to be optimum for a combined primary and secondary file system. As the file system block size is adjusted in a combined file system, 512 MB is where we began to see I/O issues.
4. Pooling in Lustre for the purpose of separating primary and secondary provides no performance improvement.
5. Bottom line, keep things simple when designing your parallel file system for SeisSpace primary and secondary. The newest hardware and file system technology solves many of the old problems.

Upcoming SeisSpace webinar

If you would like to hear more detail on the testing we did, please join us for a free webinar on May 16 at 7 a.m. Pacific/10 a.m. Eastern. Dan Grygier and I will be going over our testing and results in detail.

Register here: “SeisSpace I/O Profiling and Parallel File System Benchmarks.”

The post SeisSpace and Parallel File Systems: What are You Using? appeared first on Cray Blog.