OpenGL port - help needed

[JohnJack]

I came across the following in NVIDIA's ARB Vertex Program tutorial presentation. On page 54, the following is listed:

• When a generic attribute is specified using glVertexAttrib*(), the current value for the corresponding conventional attribute becomes undefined
  • Also true for the converse

Page 56 lists various conventional attributes (e.g. vertex.position, vertex.color), and their generic counterparts (vertex.attrib[0], vertex.attrib[3], respectively).

This would imply that when using deprecated features (e.g glColor4f, glColorPointer), the vertex program must reference conventional attributes (vertex.color), whilst attributes assigned by glVertexAttribPointer would reference the specific attribute index (vertex.attrib[]) to which they have been assigned. The only exception is vertex.attrib[0], which is aliased to vertex.position (according to the OpenGL 2 spec).

---

Last edited by JohnJack; March 2nd, 2011 at 19:00..

[patrickvl]

Originally Posted by JohnJack

This would imply that when using deprecated features (e.g glColor4f, glColorPointer), the vertex program must reference conventional attributes (vertex.color), whilst attributes assigned by glVertexAttribPointer would reference the specific attribute index (vertex.attrib[]) to which they have been assigned.

Oh man! When will the hurting stop?

I was hoping that we would need to recompile each shader only once, but now we already have two variations :
- in cases where the NV2A seems to apply the mvp itself (without any mention of that in the shader), we have to append that rotation ourselves
- when a shader is used in immediate mode rendering, the attributes must be conventional - when we're rendering from pointers it should specify generic attributes.

I was thinking - instead of using glVertexAttribPointer, maybe we should use the (deprecated) conventional gl*Pointer functions as much as possible, so that the shader can be used for both rendering methods. (I really don't want to generate different versions of a shader for difference usage-scenarios). Any objections against this?

PS: Another solution could be to do what the D3D version of Cxbx and Dxbx does : Collect all immediate mode vertex data into a buffer and render that with one call. The advantage of that is, that everything can be drawn using glVertexAttribPointer (and thus allows us using just generic shader attributes), but it would require a bit of housekeeping - as only at the end of the data we know for sure which attributes where supplied.
This brings me to another question: I've seen shaders that use attributes that where not part of a stream - am I correct in assuming that the values set to these attributes using glVertexAttrib stay the unmodified per-vertex and can thus be seen as a few more constants?

__________________
Dxbx dev - Delphi FTW ;-)

---

Last edited by patrickvl; March 2nd, 2011 at 21:47..

[JohnJack]

Originally Posted by patrickvl

I was thinking - ... Any objections against this?

If using gl*Pointers is easier to maintain than using glVertexAttrib*, then by all means use it. I'm biased towards generic attributes. Maintaining 1 shader is better than 2. However, I don't understand the following:

... everything can be drawn using glVertexAttribPointer (and thus allows us using just generic shader attributes), but it would require a bit of housekeeping - as only at the end of the data we know for sure which attributes where supplied.

Is this the case with all rendering modes (immediate mode, vertex buffers), where the 'push buffer' sends you a heap of memory and on its tail is a description of what the data represents?

I've seen shaders that use attributes that where not part of a stream - am I correct in assuming that the values set to these attributes using glVertexAttrib stay the unmodified per-vertex and can thus be seen as a few more constants?

Is this behaviour defined in Direct3D8 for XBox? It sounds as though the attributes were part of the current state. The glVertexAttrib spec states:

The value of each generic vertex attribute is part of current state, just like standard vertex attributes, and it is maintained even if a different program object is used.

This means that execution order is important.

[patrickvl]

Originally Posted by JohnJack

Is this the case with all rendering modes (immediate mode, vertex buffers), where the 'push buffer' sends you a heap of memory and on its tail is a description of what the data represents?

When we're dealing with vertex-buffer rendering, the push-buffer receives commands that set the vertex attribute pointers, so there's not much to it, really. (There's a caveat here, but I'll talk about that later)

However in immediate mode, when the Begin command is received we don't know what kind of data setters will follow until we've received the End command. What I meant was, that we could collect all vertex data send to us in between those two calls, create buffers for each type, and draw that using vertex attribute pointers, so that we can suffice with just the vertex shader variant that's using generic attributes.

Originally Posted by JohnJack

Is this behaviour defined in Direct3D8 for XBox?

The SDK docs don't mention this - but one of the samples (CompressedVertices) uses a vertex shader that reads v9 (Texcoords) without any mention of this in the sample code. (Although to be honest, this sample doesn't even use textures, so it might well be an oversight in the shader.) But the quote you gave on glVertexAttrib does indicate that attributes set with that API (and not supplied via a pointer) maintain their value, so they can indeed be seen as a constant.


About this caveat : The Xbox1 D3D implementation supports a few NV2A specific data types that are not available on the PC version of D3D8 (nor D3D9); In Dxbx we inherited code from Cxbx that converted those types to types that are supported;

• D3DVSDT_NORMSHORT1 is converted to D3DVSDT_FLOAT2
• D3DVSDT_NORMSHORT2 is converted to D3DVSDT_FLOAT2 (not for D3D9, as it does support D3DVSDT_NORMSHORT2)
• D3DVSDT_NORMSHORT3 is converted to D3DVSDT_FLOAT4
• D3DVSDT_NORMSHORT4 is converted to D3DVSDT_FLOAT4 (not for D3D9, as it does support D3DVSDT_NORMSHORT4)
• D3DVSDT_NORMPACKED3 is converted to D3DVSDT_FLOAT3
• D3DVSDT_SHORT1 is expanded to D3DVSDT_SHORT2
• D3DVSDT_SHORT3 is expanded to D3DVSDT_SHORT4
• D3DVSDT_PBYTE1 is converted to D3DVSDT_FLOAT1
• D3DVSDT_PBYTE2 is converted to D3DVSDT_FLOAT2
• D3DVSDT_PBYTE3 is converted to D3DVSDT_FLOAT3
• D3DVSDT_PBYTE4 is converted to D3DVSDT_FLOAT4
• D3DVSDT_FLOAT2H is converted to D3DVSDT_FLOAT4

Any idea how these should be mapped to OpenGL? (I suspect there's much less conversion needed, but it all depends on the available card capabilities.)

__________________
Dxbx dev - Delphi FTW ;-)

[JohnJack]

Originally Posted by patrickvl

What I meant was, that we could collect all vertex data send to us in between those two calls, create buffers for each type, and draw that using vertex attribute pointers, so that we can suffice with just the vertex shader variant that's using generic attributes.

Why create buffers for each type, when the gl*Pointer functions allow you to read from interleaved data?

But the quote you gave on glVertexAttrib does indicate that attributes set with that API (and not supplied via a pointer) maintain their value, so they can indeed be seen as a constant.

When it said just like standard attributes, they were referring to conventional attributes. Both are saved. It is similar to using one glColor call to set the color of all glVertex calls that follow.

About this caveat : The Xbox1 D3D implementation supports a few NV2A specific data types that are not available on the PC version of D3D8 (nor D3D9);...

OpenGL is well known for allowing allowing different data types to set conventional attributes (e.g. glColor4ub, glColor4i, glColor4s ...). Internally, they are converted to a float point number when accessed in the shader. True integer support only arrived with the advent of D3D10/OpenGL3-level hardware.

The glVertexAttribPointer spec states:

... type specifies the data type of each component, and stride specifies the byte stride from one attribute to the next, allowing vertices and attributes to be packed into a single array or stored in separate arrays. If set to GL_TRUE, normalized indicates that values stored in an integer format are to be mapped to the range [-1,1] (for signed values) or [0,1] (for unsigned values) when they are accessed and converted to floating point. Otherwise, values will be converted to floats directly without normalization.

So you can continue to do the conversion yourself (as you've done for D3D9), or let OpenGL do it for you. With the exception of NORMPACKED3 (I think) and FLOAT2H, you just have to set the relevant parameters of glVertexAttrib(Pointer) to handle your data.

The documentation for these functions has all of the information you need.

---

Last edited by JohnJack; March 3rd, 2011 at 10:49..

[patrickvl]

Originally Posted by JohnJack

Why create buffers for each type, when the gl*Pointer functions allow you to read from interleaved data?

Because in immediate mode rendering, the vertex-data is delivered via SetVertexData calls and pushed directly into the push-buffer. So we cannot 'just' set a pointer and a stride, as the calls can be ordered differently per vertex. Also, some components (like colors or point-sizes) could appear less frequently than the number of vertices being drawn.

So *if* we're going to convert immediate-mode rendering to buffer-based rendering (to simplify the vertex shader recompiling), it would mean building up buffers for each attribute that's written to between the first and last vertex position (attribute values send prior to the first vertex position can be set directly using glVertexAttrib of course, so these won't need a buffer).

Also note in this scenario that all changing attributes will need to have a specific value alongside each vertex position, or else the buffers won't line up anymore. This implies lots of repeating values (especially for attributes that don't change all that much) - sure it's overhead, but it's the only way to get this working. (It's either that, or juggling with various vertex shader variants.)

Originally Posted by JohnJack

When it said just like standard attributes, they were referring to conventional attributes. Both are saved. It is similar to using one glColor call to set the color of all glVertex calls that follow.

Got it, thanks.

Originally Posted by JohnJack

With the exception of NORMPACKED3 (I think) and FLOAT2H, you just have to set the relevant parameters of glVertexAttrib(Pointer) to handle your data.

Yeah, the flexibility in this regard is one of the reasons that OpenGL is preferable to D3D. (I did look into your statement that D3D9 is also capable of pointer-based vertex stream buffers, but I couldn't corroborate that - as far as I can tell, you still need to make a copy - something I'd rather avoid.)

By the way, if we decide to keep immediate-mode rendering (and thus don't collect vertex data values into buffers) the unsupported data-types (NORMPACKED3 and FLOAT2H) won't bite us, as the various SetVertexData calls don't support these types (pfew!)

__________________
Dxbx dev - Delphi FTW ;-)

[JohnJack]

Originally Posted by patrickvl

I did look into your statement that D3D9 is also capable of pointer-based vertex stream buffers, but I couldn't corroborate that - as far as I can tell, you still need to make a copy - something I'd rather avoid.

Yes, a copy to a dynamic vertex buffer would be required (similar to VBOs in OpenGL). But I never saw copying data to GPU/AGP memory as an issue. I was referring to the techniques discussed on this page:
Programming One or More Streams (Direct3D 9) (Windows)

Btw, converting all of your rendering code to use conventional attributes would be easier. You would just have to modify your shader to use conventional attributes.

---

Last edited by JohnJack; March 3rd, 2011 at 13:44..

[patrickvl]

Oh, it's not the copying, but the accompanying conversions that I want to avoid!

For example, the 'Cartoon' sample (showing a yellow cartoon-shaded Robot) dropped from 60 fps to 7 fps when I started converting D3D vertex buffers on every frame. I had to do that in my 'less patching' approach (based on D3D8), which is on-hold right now since I'm researching this push-buffer emulation using OpenGL. Sure, a CRC based cache can alleviate that of course, but I'd rather switch over to OpenGL entirely.

__________________
Dxbx dev - Delphi FTW ;-)

[patrickvl]

Originally Posted by JohnJack

converting all of your rendering code to use conventional attributes would be easier. You would just have to modify your shader to use conventional attributes.

Could you explain why conventional attributes are easier? As I understand, conventional setting attributes would at least require the correct call for each attribute (glColorPointer, glNormalPointer, glVertexPointer, etc.) to set vertex data pointers with. Using glVertexAttribPointer sounds easier...

Or are you referring to the fact that immediate-mode drawing requires no buffering if we use conventional attributes in the recompiled shader? I can see the logic in that, but there's one little problem to solve in that approach:
Unmapped attributes;

For most vertex attributes, there is a conventional setter, but vertex.attrib[6] and [7] have no setter. Also, if the card only supports 4 texture channels, vertex.attrib[13], [14] and [15] are also not accessible using conventional setters. And then there's the restriction on allowed GL calls between glBegin/glEnd :

The only GL commands that are allowed within any Begin/End pairs are the commands for specifying vertex coordinates, vertex colors, normal coordinates, texture coordinates, generic vertex attributes and fog coordinates (Vertex, Color, SecondaryColor, Index, Normal, TexCoord and MultiTexCoord, VertexAttrib*ARB, FogCoord)

Perhaps we should divert to generic attributes when a slot is not covered by a conventional attribute? (Surely they can be used alongside each other in one and the same shader?)

__________________
Dxbx dev - Delphi FTW ;-)

[patrickvl]

Wait a minute - there's another solution possible too! Instead of going with conventional shader inputs, just to support immediate mode rendering, we could instead use generic attributes by mapping the immediate mode data onto generic attributes, so that a glColor(...) call becomes a glVertexData(3, ...) call, etc. That would allow all inputs, still takes only one vertex shader variant and wouldn't require a buffer for immediate mode rendering! Would that be possible/wise?

__________________
Dxbx dev - Delphi FTW ;-)

[JohnJack]

we could instead use generic attributes by mapping the immediate mode data onto generic attributes, so that a glColor(...) call becomes a glVertexData(3, ...) call, etc

Yes, glVertexAttrib will work, as it can be called between glBegin/glEnd. Great idea ! The flexibility of OpenGL is one of the reasons many implementations are generally slower than their Direct3D counterparts.

Regarding your earlier question.

Originally Posted by ARB Vertex Program spec

Applications that mix conventional and generic vertex attributes in a single rendering pass should be careful to avoid using attributes that may alias. To limit inadvertent use of such attributes, loading a vertex program that used a pair of attributes that may alias is guaranteed to fail. Applications requiring a small number of generic vertex attributes can always safely use generic attributes 6 and 7, and any supported attributes corresponding to unused or unsupported texture units. For example, if an implementation supports only four texture units, generic attributes 12 through 15 can always be used safely.

So it is possible to mix generic attributes with conventional ones, as long as they don't alias one another.

Multitexturing isn't as simple using TexCoordPointer as it is with using VertexAttribPointer, as you have to specify the active texture unit that uses your texture coordinates. If an implementation supports EXT_direct_state_access, MultiTexCoordPointerEXT could be used to alleviate that problem. However, that extension is not supported by current legacy products from AMD or NVIDIA (pre-Geforce 6 and pre-Radeon HD cards), nor is it supported by Intel. Anyway, it is unlikely that your emulator will run on a GPU whose OpenGL implementation cannot handle 8 texture coordinate sets.

One conventional attribute that isn't supported on most graphics cards is vertex.weight, as few cards (atleast on Windows) support ARB_vertex_blend. Thus it can only be set using the attribute pointer (+1 for generic attributes).

The main benefit of conventional attributes is that it improves the readability of the code. Generic attributes are forward compatible (supported in the core profiles of OpenGL 3+), although this has little meaning if deprecated functionality continues to be supported (not a bad thing).

---

Last edited by JohnJack; March 3rd, 2011 at 18:25..

[patrickvl]

Oops, I just realised that using glVertexAttrib will probably not work with the fixed function pipeline!
I fear we'll have to settle for a mixed shader : All conventional attributes as-is, all other slots using generic attributes, even though it's not advised (as stated page 56 here).

__________________
Dxbx dev - Delphi FTW ;-)

[JohnJack]

Originally Posted by patrickvl

Oops, I just realised that using glVertexAttrib will probably not work with the fixed function pipeline!

Err ... weren't you planning on using shaders to emulate the FFP? The benefit of using conventional attributes was that it allowed shaders access to values that were set by the FFP. Btw, Example Program 2 (pages 50-51) shows how to emulate fixed function lighting.

---

Last edited by JohnJack; March 3rd, 2011 at 19:02..

[patrickvl]

Originally Posted by JohnJack

Err ... weren't you planning on using shaders to emulate the FFP? The benefit of using conventional attributes was that it allowed shaders access to values that were set by the FFP. Btw, Example Program 2 (pages 50-51) shows how to emulate fixed function lighting.

In due time, yes, but it's a can of worms I don't want to open up yet. (And besides, first I have to discover how the NV2A fixed function pipeline actually works - what steps are done when exactly).

Actually, it's quite simple to make the code conditionally compile either to use generic attributes or conventional attributes; I've done most of the work already

Edit : Here's the final switch. As I said : not much code.

__________________
Dxbx dev - Delphi FTW ;-)

---

Last edited by patrickvl; March 3rd, 2011 at 19:40..

[JohnJack]

Originally Posted by patrickvl

first I have to discover how the NV2A fixed function pipeline actually works - what steps are done when exactly.

Maybe blueshogun can provide some insight, as he has some screenshots on his website with Cxbx using D3D8. Another source of information is the WINE project, which has spent the last decade layering D3D8/9 on top of OpenGL, so their codebase should provide more insight into translating D3D8 to OpenGL (minus the differences that are specific to the NV2A).

EDIT: On looking at your code, I noticed that you commented out the glScalef call that is required to make the coordinate system of OpenGL left-handed. The transformation shader does not take this into account either. Is this handled elsewhere in your code? If you focus on pure fixed function vertex processing, where you know the inputs, and the expected output, you should be able to make the necessary transformations to the contents in the push-buffer to make the result appear on the screen.

---

Last edited by JohnJack; March 4th, 2011 at 00:42..

[HackerNoobKid]

JohnJack is right about WINE project, they really have good info about layering D3D8/9 on top of OpenGL. It is good a way of obtaining information about it

[Shock120]

Could asking around on OpenGL.org help?

About the OpenGL.org website

OpenGL.org is a vendor-independent and organization-independent web site that acts as one-stop hub for developers and consumers for all OpenGL news and development resources. It has a very large and continually expanding developer and end-user community that is very active and vested in the continued growth of OpenGL. There are discussion boards, news groups and a variety of other venues for learning how to code using OpenGL, getting feedback on your projects, job opportunities, and support for the consumer trying to play a 3D game

I hope there are people nice enough to help.

---

Last edited by Shock120; March 4th, 2011 at 17:09..

[JohnJack]

I saw in Revision 1619 of your push buffer code:

Disabled conventional vertex attributes, as we will probably be using a shader instead of the fixed function anyway.

Yaay . Plus, you also imported some stuff from WINE. Is it working in the way you expect?

---

Last edited by JohnJack; March 5th, 2011 at 15:10..

[patrickvl]

Well, yes and no;

If I reproduce the 'Vertices' SDK sample in a test app, the wined3d 'transform_projection' code works for both immediate and deferred rendering (with a simple shader) and for immediate mode (without a shader). (Deferred rendering without a shader doesn't show up, I assume that's only logical because the fixed-function pipeline doesn't handle generic vertex attrib pointers.)

However, when I run the Vertices sample inside Dxbx I still see nothing. The problem is, that I had to alter a few things in the test app to get things rendering:
- The 'SuperSampleScale' z component send to the Xbox push-buffer is 16777215, which needs to be scaled down to 1 to get something on-screen.
- Also, the shader needs to be extended with a mvp transformation (although I wouldn't expect that to be necessary...)
On the bright side, the projection-view matrix used by the Vertices sample when running inside Dxbx is identical to to one in the test-application, so that part seems to be good now.
WineD3D does alter ARB shaders too, so I hope to find the issue sooner rather than later ;-)

__________________
Dxbx dev - Delphi FTW ;-)

[JohnJack]

Originally Posted by patrickvl

- The 'SuperSampleScale' z component send to the Xbox push-buffer is 16777215, which needs to be scaled down to 1 to get something on-screen.

Did this sample work in your D3D implementation?
I noticed on the previous page that you had:

NV2A > SetVertexShaderConstant(0, {1, 1, 16777215, 1}{0.53125, 0.53125, 0, 0})

Does the Vertices sample make this call, or the D3D runtime internally? Try changing the far clip plane in your Frustum/Ortho calls to this value.
e.g.

glMatrixMode(GL_PROJECTION);
glOrtho (0,640,480,0,0,16777215);

- Also, the shader needs to be extended with a mvp transformation (although I wouldn't expect that to be necessary...)

Yes, that would be required as D3D's near and far clip plane is different from OpenGL's ... which maybe is why WineD3D alters the shaders as well (just guessing).

---

Last edited by JohnJack; March 5th, 2011 at 18:50..