Author Topic: Mass-exporting prefabs as DataNodes  (Read 14585 times)

Bradamante3D

  • Jr. Member
  • **
  • Thank You
  • -Given: 2
  • -Receive: 0
  • Posts: 79
    • View Profile
Mass-exporting prefabs as DataNodes
« on: March 24, 2016, 05:28:22 PM »
Hi there,

I just recently updated to TNet 3, I am now going through the examples and so far I like what I am seeing.

To enable modding I am using asset bundles. That works fine as relatively simple since I don't plan to have server-side mods that people would have to download. Instead there's a mod filter for the server browser.
Going from Unity 5.3.x to 5.4b I noticed that the creation of asset bundles now takes a build target. I find that tedious since I am targeting Win/Mac/Linux. So let's say that's my motive to go the DataNode route.

My asset bundles can be devided into three categories: text files, prefabs and textures (png files). Each of these contains multiple asset bundles with A LOT of objects. I assume I can export prefabs to DataNodes one by one manually. Is there a less tedious way? And what about text files and pictures?

Is there a way to translate my asset bundle process to DataNode creation?
#301224014, #301432336, #302399130

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,128
  • Toronto, Canada
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #1 on: March 27, 2016, 03:18:48 AM »
Assuming you have a list of game objects you want to export, such as by selecting them in inspector, you can iterate through them and export each one easily:
  1. var gos = UnityEditor.Selection.gameObjects;
  2. foreach (var go in gos) go.Serialize().Write(go.name + ".bytes", DataNode.SaveType.Compressed);
Same can be done with textures, except in their case you specify a node to serialize into:
  1. DataNode node = new DataNode("My Texture");
  2. tex.Serialize(node);
Since DataNode is a tree-like structure, you can just save all your texures in a single DataNode like so:
  1. DataNode root = new DataNode("Textures");
  2. foreach (var tex in textures) tex.Serialize(root.AddChild(tex.name));
  3. root.Write("textures.bytes", DataNode.SaveType.Compressed);
Same goes for game objects btw... you dont need to save each individual game object in their own file. You can just serialize them using the gameObject.Serialize() method and add the resulting DataNode as a child of another DataNode. When de-serializing, simply GetChild() and node.Instantiate() it, converting it to an instance of your original prefab.

Write:
  1. DataNode root = new DataNode("Prefabs");
  2. var gos = UnityEditor.Selection.gameObjects;
  3. foreach (var go in gos) root.children.Add(go.Serialize());
  4. root.Write("prefabs.bytes", DataNode.SaveType.Compressed);
Read:
  1. DataNode node = DataNode.Read("prefabs.bytes");
  2. foreach (var child in node.children)
  3. {
  4.     Debug.Log("Prefab: " + child.name);
  5.     // Instantiate any prefab you like:
  6.     GameObject go = child.Instantiate();
  7. }
P.S. If you want to know more, open the DataNodeExporter.cs file and have a look inside. All the file import/export functions are there.
« Last Edit: March 27, 2016, 03:24:33 AM by ArenMook »

Bradamante3D

  • Jr. Member
  • **
  • Thank You
  • -Given: 2
  • -Receive: 0
  • Posts: 79
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #2 on: March 27, 2016, 11:38:02 AM »
I am also bundling GameObjects with ParticleSystem components into Asset Bundles. In the past you said that TNet was not able to serialize those since Unity did not expose the PS to scripting. At this point Unity should do that to my knowledge.

So, with current versions of Unity (5.3? 5.4?) can the current or near future versions of TNet serialize ParticleSystems?

What about custom shaders? I am using the Unity Standard shader(s) mostly, but sometimes I use the Sonic Either Bloom shader(s).
« Last Edit: March 27, 2016, 11:44:26 AM by Bradamante3D »
#301224014, #301432336, #302399130

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,128
  • Toronto, Canada
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #3 on: March 27, 2016, 06:09:55 PM »
Sonic Bloom is awesome. Using it myself.

In theory, you can serialize the shader's text and then create it at run-time, but it will only work for vert+frag shaders, not surface shaders.

I am not sure about particle systems. If Unity exposed the fields, then yes, it will work. Try it. Keep in mind you can bring your prefabs into the scene and put other prefabs as children. If those prefabs are in the Resources folder, TNet will reference them instead of saving their data -- so you can have nested prefabs, basically.

Bradamante3D

  • Jr. Member
  • **
  • Thank You
  • -Given: 2
  • -Receive: 0
  • Posts: 79
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #4 on: March 28, 2016, 11:54:24 AM »
Uhm let's keep the conversation on a level even my brain can process. I am using the SE/Emissive/Textured shader on a lot of content. In those setups I don't include a texture though, it's just color-based. Can DataNode serialize that or not?

If it doesn't that would make things kinda useless for me. Now I am screwed because I don't like Unity's build target requirement either.
#301224014, #301432336, #302399130

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,128
  • Toronto, Canada
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #5 on: March 30, 2016, 12:59:41 AM »
Keep the shader in the Resources folder, and TNet will be able to serialize everything just fine. It will see that the shader exists as part of your resources, and it will include a reference to it rather than including the shader's code/text.

phoenix

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 3
  • Posts: 49
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #6 on: March 31, 2016, 12:27:54 AM »
Would you consider adding an encryption type to the serialization so that the .bytes are saves encrypted?

This is how unity recommends protecting asset bundles
http://docs.unity3d.com/Manual/protectingcontent.html

also, they might have added it natively to their asset bundles, I cannot tell from this doc.
http://docs.unity3d.com/Manual/AssetBundleCompression.html


ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,128
  • Toronto, Canada
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #7 on: March 31, 2016, 04:06:10 AM »
That's something that should take place outside of TNet, as including encryption as a part of the kit would just mean that everyone can break it. If you look at the Unity's page you linked, they suggest exactly that.

When you serialize something with TNet, you get a byte[] array. Instead of just writing it out to a file, run your encryption code on it, and then save it. Do the opposite for loading it -- read the file, decrypt it, then feed it to DataNode.Read().

phoenix

  • Newbie
  • *
  • Thank You
  • -Given: 0
  • -Receive: 3
  • Posts: 49
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #8 on: March 31, 2016, 11:16:12 AM »
Is it possible to take just the parts that do the data node serialization and use them with Tnet 2.1.1?

I cannot upgrade my projects to Tnet 3 yet because of customizations I did to Tnet 2.1.1 but I want to use the new Data Node features.

What files would I need to use for it?

toga94

  • Newbie
  • *
  • Thank You
  • -Given: 8
  • -Receive: 0
  • Posts: 30
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #9 on: April 01, 2016, 11:49:10 AM »
you can use  [RFC] , playfab or with mysql

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,128
  • Toronto, Canada
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #10 on: April 01, 2016, 12:13:18 PM »
Is it possible to take just the parts that do the data node serialization and use them with Tnet 2.1.1?

I cannot upgrade my projects to Tnet 3 yet because of customizations I did to Tnet 2.1.1 but I want to use the new Data Node features.

What files would I need to use for it?
You can email me with your OR# and I can send you the last backwards-compatible build of TNet 2 that has many of the features of TNet 3 in it, including the serialization.

Bradamante3D

  • Jr. Member
  • **
  • Thank You
  • -Given: 2
  • -Receive: 0
  • Posts: 79
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #11 on: April 09, 2016, 04:45:57 PM »
Uhm, no. No success here. I saved a prefab as binary. Read it back and the console says:

  1. Unable to resolve type 'Rendering.ReflectionProbeUsage'
  2. UnityEngine.Debug:LogError(Object)
  3. TNet.Serialization:NameToType(String) (at Assets/TNet/Common/TNSerializer.cs:296)
  4. TNet.Serialization:ReadType(BinaryReader, Int32&) (at Assets/TNet/Common/TNSerializer.cs:2342)

The reflection probe or light probe setting on the Mesh Renderer is Off. The shaders of the materials are all the Unity Standard shader. On instantiating all the children (particle systems) are gone :(

That's build in Unity stuff, so why would it not work?

EDIT: When I search the docs of Unity 5.4b the term "ReflectionProbeUsage" is not found. Is that a 5.4b vs 5.3 thing?

EDIT2: Reading the same prefab back as an exported text file I get a different error:

  1. Unknown prefix: 73 at position 1
  2. UnityEngine.Debug:LogError(Object)
  3. TNet.Tools:LogError(String, String, Boolean) (at Assets/TNet/Common/TNTools.cs:1001)
  4. TNet.Serialization:ReadObject(BinaryReader, Object, Int32, Type, Boolean) (at Assets/TNet/Common/TNSerializer.cs:2724)

That can't be right?

Read:
  1. DataNode node = DataNode.Read("prefabs.bytes");
  2. foreach (var child in node.children)
  3. {
  4.     Debug.Log("Prefab: " + child.name);
  5.     // Instantiate any prefab you like:
  6.     GameObject go = child.Instantiate();
  7. }
« Last Edit: April 09, 2016, 05:20:55 PM by Bradamante3D »
#301224014, #301432336, #302399130

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,128
  • Toronto, Canada
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #12 on: April 10, 2016, 10:51:55 PM »
Interesting... doing this:
  1. Debug.Log(typeof(UnityEngine.Rendering.ReflectionProbeUsage));
gives:
  1. UnityEngine.Rendering.ReflectionProbeUsage
Yet when I do this:
  1. var type = System.Type.GetType(typeof(UnityEngine.Rendering.ReflectionProbeUsage).ToString());
The result is "null". This must be some Unity glitch. You can fix it by adding a special case to TNSerializer.cs, NameToType() function around line 281:
  1. #if !STANDALONE
  2.                         else if (name == "Rendering.ReflectionProbeUsage")
  3.                         {
  4.                                 type = typeof(UnityEngine.Rendering.ReflectionProbeUsage);
  5.                         }
  6. #endif

Bradamante3D

  • Jr. Member
  • **
  • Thank You
  • -Given: 2
  • -Receive: 0
  • Posts: 79
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #13 on: April 12, 2016, 04:28:19 PM »
What version did you use for testing? I am on 5.4b13 (and I won't go back to 5.3.x). It propably got lost in my last posting, but the docs for 5.4 don't know of this ReflectionProbeUsage attribute, only the 5.3 docs do. Is this a 5.3 vs 5.4 thing?

EDIT:

I tested your code snippet. Now the error message is gone, but the children of the GameObject are still skipped. Does DataNode not serialize children automatically? I would have expected so?
Also the error still remains when I import a text-based node (instead of binary) and the GameObject is not recreated at all.

EDIT2:

I did some more testing and had some interesting results. Instead of using my own assets I simply exported a default particle system. When I load the DataNode back I got this bug, where the loaded particle system is there, but the material is lost so everything is pink. In that link, read my oh-so-delightful exchange with Unity support on this. Here I described my workaround.

Workaround 1: To export from OpenGL Core -> OpenGL Core on Mac, force-include the built-in shaders in the Graphics settings. This was for asset bundles workflows on Mac OS X with Unity 5.3.x. Now the thing seems back with DataNodes on 5.4b? Including shaders in the Graphics settings does not help. Workaround 2: Force OpenGL 2. These workarounds do not improve things with Unity 5.4 and DataNodes.

When I export the particle system as text the material related lines say:

  1. sharedMaterial = "asset|Material|unity_builtin_extra"
  2.                         sharedMaterials = string[]
  3.                                 asset|Material|unity_builtin_extra

As a next step, I will test asset bundle import/exports. Would be too funny if they not only did not fix stuff in 5.4, but even killed the workaround.
« Last Edit: April 12, 2016, 05:25:08 PM by Bradamante3D »
#301224014, #301432336, #302399130

Bradamante3D

  • Jr. Member
  • **
  • Thank You
  • -Given: 2
  • -Receive: 0
  • Posts: 79
    • View Profile
Re: Mass-exporting prefabs as DataNodes
« Reply #14 on: April 12, 2016, 05:19:46 PM »
.
#301224014, #301432336, #302399130