Although I can’t disclose the actual code i’m using right now, as I wish to release some class packages for unity and flash later this year which should be available for all and in the interim want to use it for commecial purposes, however I can point you in the right direction in terms of structure and reference code material if that would be of help?

Congratulations man !!
I tried the javascript and sockets route myself, but got stumped on the flash policy stuff.
Hopefully your development will progress smoothly.
Looking forward very much to your planned release.

Ciao, Patrick

Thankyou for the feedback, most appreciative, as for the policy stuff, I can share this part of the code which should hopefully help others, although I am looking into making this more efficient and into a class. This should make sense to those in the know about threads and listeners:-

In Flash send webcam data using localhost, I used port 8080, in Unity use TCP listener localhost same port, in your thread in the while listening loop,

Code:

mySocket = tcp_Listener.AcceptTcpClient();
TcpClient(Host,Port);
stream = mySocket.GetStream();
bytesReceived = 0;
receivedBytes =0;
if (setpolicy) {
  byte[] flashpolicy = System.Text.Encoding.ASCII.GetBytes(“<cross-domain-policy><allow-access-from domain=\”*\” to-ports=\”*\”/></cross-domain-policy>”);
   while(shouldRun) {
        byte[] bytes = new byte[mySocket.ReceiveBufferSize];
       receivedBytes = 0;
       receivedBytes = stream.Read (bytes, 0, (int mySocket.ReceiveBufferSize);
       if (receivedBytes>2) {
           stream.Write(flashpolicy, 0, flashpolicy.Length);
           setpolicy=false;
           break;
       }
       if(receivedBytes == 0) {
           break;
       }
   }
} else{
   // Read your image bytes and process into texture
}

Essentially flash’s first communication to unity will be a string asking for a policy ‘<policy -file-request/>’, it first asks on port 843 then the port you’ve tried to open, if you reply with the string above, it starts to send your data across otherwise blocks flash from communicating.

Hope this helps, Jon

Thanks for the info.
I must say that my video transfer/augmented reality project is on the backburner at the moment.
But I’ll give you a shout when I make progress again.

You did not mention anything about compression soo.. afaik you get raw bitmap data from the webcam. Before you send it over the socket, compress the data using something “fast” (perhaps run-length encoding?) as3corelib (http://code.google.com/p/as3corelib/) includes jpeg and png encoders, but i think it might be slower than sending compressed raw data since there is also the overhead of decoding.

Very interesting, I look forward to seeing Unity being used for AR!

On this version, no compression is included JPG/PNG encode it tended to slow things down. The image transmitted is unfortunatly low-resolution currently in byte format, and I haven’t tried pushing up the resolution yet due to other work commitments. If higher resolutions fail performance speed some kind of vector math based compression could be utilised, this works quickly in as3, and some of the routines I wish to push into the AS3 class such as background removal, motion detection and OpenSURF may take away some of the needs of transmitting all colour image data.

This all remains to be seen although one thing I am sure of, is that when I release the first draft i’m hoping the community will overhaul and help with optimisation. What would the feelings be on setting up a SVN Depository with GoogleCode in the near future?

This is a pretty brilliant idea… I’ve done a lot of AR and would love to marry Unity’s capabilities with that. Flash is so limited it’s painful. I definitely need to look into this more. Thanks for sharing some ideas and techniques.

Hello Jon,

Well done!

I’m trying very hard to understand what’s going on here. Could you point me in the right direction as you offered to Daniel above? Any help would be hugely appreciated!

Many thanks in advance,

-r

Although I have carried out further experimentation, at higher resolutions and with splitting data over many sockets as opposed to one, the best results still seem to come from the example shown, although I have further tests to do and scripts to ammend including a flash control script this is still ongoing in my spare time.

Both flash SWF and Unity3d objects must be loaded in browser on webpage or (possibly both pages active at same time, this I haven’t tried)

The described process below is ideal for taking a snapshot of image data from flash and sending it to unity, currently this works best at sending 160×120 pixels RGBA which is very low res, but probably ok for motion tracking and some AR techniques in terms of speed, even for simple facial texture mapping for game characters, but there is no reason this couldn’t be scaled, however as it is web-based it is important to consider the speed of users machines.

The Current Working ActionScript code looks something like (pseudo-code):

SetupCamera, Bitmap (320×240)
Set up socket (localhost,port 8080)
Set up Connect timer,started
Set up SendData Timer,stopped
Connect Timer Function {Connect}
On Connect Init Listener{Send Data Timer Start, Stop Connect Timer}
On Close Connect Listener{Reset Timers, Re-Connect}
On Errors Listener{try..Catch then Close Connect(null) – Re-Connect}
Send Data Timer Function{if not connected connect, get Image from cam, scale using matrix (160×120), get bytes of bitmapdata, send data }

The current Unity.cs script looks like (pseudo-code):

Declare vars: texture2d,pixels color array, material, thread, TcpClient, is running bool, sent policy bool etc
Start: Set up thread function, Put pixel data to chosen material, thread function starter
LateUpdate: Copy any pixelsdata[] to texture, Update
Thread (see forums on thread scripts): {
Try Setup 8080 listener, Try GetClient Get Stream, While !Pending Sleep else
if (connected and data, haven’t sent policy), send policy(see above), set policy sent bool true,
if (connected and data, have sent policy), Read Bytes from stream (160x120x4) if complete process bytes to pixeldata
Catch all errors, solve problems.
OnDisable: (important otherwise web-browser crash when leave site): Close Connection, Close Thread, Reset Vars
OnAppQuit: Close Connection, Close Thread, Reset Vars

Thats about as far as I can go for now, and this should have given enough information, including the posts above for coders to get this working, Hope this helps.

By the way, I stumbled across a cunning way to get the memory address of flash bitmap data, but due to sandboxing and security, no way of passing this to unity in webplayer plus its probably local memory adress so would require base address, if any devs have any ideas or indeed anyway of accessing the browser clipboard from unity or accesing unlocal memory address in webplayer, I would be keen to hear from you, although to my knowledge for security reasons I understand why this should not be possible.

Great article, thank you!

The demo link is not working.

Is it possible to see the webplayer demo?

big thanks again!

Hello, many thanks for your response, i’m in the process of overhauling my site, so will shift the demo onto this site as soon as I can find the last release lurking on my portable hard drive somewhere, I will change the link above in the main article and post it below this comment once I have done so that you know it’s been updated.

As an aside to this, I am aware that since Unity3d v3.0 they have embedded the same socket security policy measures as flash player to bring them both into line, possibly as a security measure (not sure whether this was my fault or part of the bigger picture). This means that my existing code may not work from V3 forward without setting up policies on the host website. I’ll be sure to report any findings and will be publishing existing code shortly on my personal site. Thanks for your patience Jon.

Hi Jon, I’m looking at sending webcam data from my own app running on the clients machine to the webplayer via sockets. What kind of speed limitations are there for that? It should be almost as quick as memory shouldn’t it, if I’m sending to/from localhost? Are the speed issues you are getting due to the data coming from flash? How does it compare to using a plugin in Unity that passes the data back through memory speed wise?

Thanks, Chris.

If you are installing your own app on the clients machine, then no need for these shennanigans, I was experimenting with this purely for allowing webcam access for augmented reality in unity through the webbrowser with no special download installation bar flash player (most people have) and unity player (increasing in popularity) which most work places should not have a problem allowing through.

It seems you may as well, just create a stand-alone unity app with a webcam plugin and pass the memory address of the texture across to your plugin and put your webcam image data to that, then you are free to do with that data what you wish in unity, possibly send out to the web via unity’s/mono’s own communication methods.

Sending image data over sockets on local host is reasonably fast with a low resolution, but no where near as fast as using a memory address, plus if you’re capturing an image bigger than 160×120 it starts to get much slower, which is where data compression has to come into play. The plug-in windows based DLL I mashed together using unity’s quick time example and theodore watsons c++ library produced great results even from a HD Firewire Camera. Obviously the downside with this is that you need to write both a MAC and PC based plugin if you wish to go cross-platform, and it requires a download and installation from the web.

Hope this helps, please let me know if this helps answer your question.

Hi John.. Thanks for that info. I’m just looking into it now. Not sure how to go about getting the memory address of the texture from the WebPlayer though.. do you know if thats possible? That would be the best way of doing things.

As it stands I have to get the data as RGB.. then expand it up to be 96 bits/pixel (3 floats/pixel), then give it to unity to create a texture from that(which in the process converts it back to RGB 24 bits/pixel, doh! )

I did some tests on localhost and can get about 2.3Gb a sec transfer on my machine, so should be quick enough. It does seem slightly faster with more threads, so probably CPU bound. Currently considering using the www class and compressing to PNG on the plugin.. that way I could reduce bandwidth and remove the whole conv-to-float->conv-to-bytes stuff.
Not sure if I can get the www class to use a different port though, as a lot of people would have LAMP servers running on their machine that would cause all kinds of problems
Definitely interested to hear if you know of any way of either a) getting the address of something in the webplayer or b) setting textures directly from an external app (not opengl based).

just realised using www is a no-go as the security only allows comms with the serer that the webplayer is running on!

Firstly I need to to know why your using the webplayer, as you cannot use plug-ins with the webplayer, I think when I understand this I can help further.

Hi all,I re-installed the demo at:-
http://www.jon-martin.com/lab/unitycam/
I’ve fixed the link in the post and apologise for the delay,
Thanks Jon

@Jon – how did you manage to set up a TCP Listner in the Unity3D webplayer .. If I try to I get ..

ERROR Listening on TCP sockets is not allowed in the webplayer

I’ll be posting the script I used for Unity 2.6 at my site http://www.jon-martin.com shortly (later this week) with both the as3 code and the unity 2.6 code.

I believe that you might have problems with this in Unity3d version 3 due to the socket policy they imposed to standardise browser security on a par with Flash player, although I will be working with this shortly to find a resolution, I believe you should have a quick look at Unity3d Socket Security Policies, it might mean adding a snippet of xml to you root hosting or looking to use something that is similar to the utiliity in the unity app folder here :
Program Files\Unity\Editor\Data\Tools\SocketPolicyServer
Please let me know how you get on, it would be useful to know what directions already work/don’t work.

Hi Jon,

Very interested in playing with your code.
Are you still able to post the script to your website? (I have searched your site but no results…)

Any data would be great,
Cheers,
Oliver

Hi Jon,

I’m extremely interested in your Standalone Unity3D plugin made with the videoInput library it would be nice to try that out as i’m working on an idea that requires video implementation in unity. Currently i dont have the time or knowledge to build a plugin myself otherwise i would. I’ve seen the quicktime example which is really awesome and i knwo quicktime can also do video input (not sure about windows) I’m looking for a starting point at the least.

As for the cross-platform using flash I think it’s a great way to implement camera functionality online also saw a similar project from rooch84. Keep up the Great Work! I’m all for the idea of a public code repository.

Thanks!

here is a link to the code: http://jon-martin.com/?p=253

Jon.

Thank u sir, for the nice information about the fact that they had appeared to take a webcam directly into Unity utilising either Adobe Flash Player and/or the Unity3d Webplayer. I should need an more information which u gathered.
———————————-
kasey

I can see a white spinning cube and I can see my webcam, but I can see no interaction between the two. (I’m using Safari in a Snow Leopard)
Any advice?