Key exchange is a bit complex to implement in OpenSSL, but not too terribly hard once you understand the silly details of MS CAPI (or Microsoft CryptoAPI). All of this falls under the MSTA (or Microsoft Trust Agreement) protocol, that is described vaguely in various slides on the net (see download.microsoft.com).

As a intial note about the protocol, all packets are in a nested key-size-data format. Will make sense when you look at it.
I believe this is officially called TLV format (type, length, value). There are two bytes for a key or type, and two bytes for the length.

First part is what I call "querying ID" of remote system. It's a small TCP packet that the server sends to the extender that looks like this:

char queryBuf[] = {
magic - this begins all media center packets
0x51,0xc6,0x74,0xc0,
key represents query buffer, I believe
0x01,0x41,
0x00,0x48,
unknown key/data
0x01,0x9c,
0x00,0x04,
0xeb,0x06,0x09,0x90,
unknown key/data
0x01,0x18,
0x00,0x04,
0x00,0x00,0x00,0x00,
destination guid (contains MAC address of client, here replaced with 0xaa - probably must generate this)
0x01,0x10,
0x00,0x10,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x02,0x00,0xaa,0xaa,0xaa,0xaa,0xaa,0xaa,
source guid (HOSTID), replaced with 0xff's
0x01,0x13,
0x00,0x10,
0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,
unknown key/data
0x01,0x39,
0x00,0x04,
0x00,0xe0,0x4c,0x00,
unknown key/data
0x01,0x9d,
0x00,0x04,
0x09,0xb4,0x32,0x20
};

The client should respond back with something like:

char queryCorrect[] = {
magic
0x51,0xc6,0x74,0xc0,
key represents query response
0x01,0x22,
0x00,0x08,
unknown key/data (error/success?)
0x01,0x18,
0x00,0x04,
0x01,0x00,0x00,0x00
};

After that, querying is over and the client (the extender) will proceed to generate and send an RSA key. Hurrah.
It'll be in a pretty ugly little format, AND it'll be broadcast to udp port 3776 (yeck), not sent in TCP.

This key is in RSAPUBKEY format, which is kind of nice because it provides a bunch of information about it. But I'll go ahead and tell you, it's an RSA public key. However, if the client has never communicated with you, it will be all zeroes! This means that you'll have to go through... RSA negotiation.

Now that you have an actual (non-zero) RSA key, you should import it into OpenSSL (which basically means reversing it to convert endianness, and using BN_bin2bn). At this point, you'll need to send the key back to confirm it. :-(

char send_keyheader[] = {
magic
0x51,0xc6,0x74,0xc0,
0x01,0x81,
0x01,0x50,
unknown
0x01,0x18,
0x00,0x04,
0x00,0x00,0x00,0x00,
destination guid (contains MAC address replaced with 0xaa - probably must generate this)
0x01,0x10,
0x00,0x10,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x02,0x00,0xaa,0xaa,0xaa,0xaa,0xaa,0xaa,
key name (in unicode): "osxbox"
0x01,0x15,
0x00,0x0e,
0x6f,0x00,0x73,0x00,0x78,0x00,0x62,0x00,0x6f,0x00,0x78,0x00,0x00,0x00,
key header
0x00,0x00,0x01,0x80,0x01,0x1c
};

Just like the client did, you're going to send this via udp port 3776, but you're going to send it DIRECTLY back to the client. Not broadcast it. It should send you a short little acknowledgement of 36 bytes. Included in those bytes are a TCP port (the last four bytes, I believe).

This port will be used in connecting to the device for rdp invitation.