How to search all the deployment logs at once?

Automated deployment for web applications and databases

How to search all the deployment logs at once?

Postby isme » Thu Dec 19, 2013 7:49 pm

Hey RG,

It's easy to search the log for a single deployment.

You can use your browser's built-in search functions once you load the log in your browser.

But can you do the same across all logs for a given project?

I'm trying to find old logs that contained a particular error message to help explain why we changed one of our deployment procedures.

I don't see any way to do it in the web interface.

I can't find the log files on the file system, or in the RavenDB database.

Maybe I'm looking in the wrong place.

Do you know a way to do this?

Happy Christmas to you all!
Iain Elder, Skyscanner
isme
 
Posts: 83
Joined: Tue Jun 12, 2012 1:49 pm
Location: Edinburgh

Postby james.billings » Fri Dec 20, 2013 4:18 pm

Hi Iain- the logs live on the agents, in this folder:

C:\\ProgramData\\Red Gate\\DeploymentAgent\\Applications\\.Agent\\Logs

What I can't quite find is where the GUID these files use as their names is stored... the 'Deployments' document in the RavenDB doesn't seem to obviously hold it.

I'll see if I can find that out next week.
james.billings
 
Posts: 1120
Joined: Wed Jun 16, 2010 11:10 am
Location: Red Gate

Postby isme » Fri Dec 20, 2013 5:13 pm

Thanks, James.

What about SQL Server deployment logs?

SQL Server deployment agents run on the RGDM host.

On my RGDM host the folder

C:\\ProgramData\\Red Gatedoes not contain a folder called DeploymentAgent.

It contains just one folder called Licenses, which is empty.
Iain Elder, Skyscanner
isme
 
Posts: 83
Joined: Tue Jun 12, 2012 1:49 pm
Location: Edinburgh

Postby james.billings » Mon Dec 23, 2013 1:50 pm

I've checked with the team, and the SQL logs never get as far as ending up on disk; they get streamed straight between the special SQL deployment agent and Deployment Manager itself.

For both Agent and SQL deployments, the logs are stored in the Raven DB as document attachments, however the Raven Studio viewer (http://localhost:10300) doesn't appear to offer any easy way to view these. You may be able to use the commandline tool to export them out though: http://ravendb.net/docs/server/administration/export-import
james.billings
 
Posts: 1120
Joined: Wed Jun 16, 2010 11:10 am
Location: Red Gate

Postby isme » Mon Jan 13, 2014 3:37 pm

Thanks, James.

Here's the Smuggler command I used:

Code: Select all
Raven.Smuggler out http://localhost:10300/ dump.raven --operate-on-types=Attachments


It finished with this message:

Code: Select all
Done with reading attachments, total: 1764


It created a file called dump.raven approximately 13MB in size.

I opened it in Notepad++.

The attachments appear to be serialized as JSON, but they don't look familiar to me.

I can't paste a complete example because it contains binary data.

Each attachment looks a bit like this (I've elided the data stream):

Code: Select all
   {
      "Data": "H4sIAAAAAAAEAOy9B2AcSZYl[...]ubBEevVOAAA=",
      "Metadata": {
        "ContentType": "text/xml"
      },
      "Key": "attachments/tasks-34/output-log"
    }


How should I decode this?

As an aside, apparently you can't view attachments in Raven Studio because the developers "don't want to encourage their use." See issue RavenDB-909 on the Hibernating Rhinos issue tracker.
Iain Elder, Skyscanner
isme
 
Posts: 83
Joined: Tue Jun 12, 2012 1:49 pm
Location: Edinburgh

Postby Mike Upton » Mon Jan 13, 2014 4:34 pm

Hi Iain,

The Data elements in those JSON objects look like they're base64 encoded. When we write the logs to the database, they're an XML string, so you should be able to get the XML back by reversing that. In C#, the code would be something like:
Code: Select all
Text.Encoding.UTF8.GetString(Convert.FromBase64String("...base 64 string from data..."))
Mike Upton

Project Manager - SQL Compare|Data Compare|Comparison SDK
Redgate Software Ltd.
Mike Upton
 
Posts: 189
Joined: Wed May 11, 2011 8:04 am
Location: Redgate

Postby isme » Mon Jan 13, 2014 7:53 pm

I was able to decode something, but it still makes no sense.

The value of the Data property is not pure base64. There is some binary stuff in there as well.

In Notepad++ it looks like this:

Image

If you ignore the the sequences of \"NUL NUL EOT ÿ û\" and copy the value to a new buffer, you get this:

H4sIAAAAAAAEAOy9B2AcSZYlJi9tynt/SvVK1+B0oQiAYBMk2JBAEOzBiM3mkuwdaUcjKasqgcplVmVdZhZAzO2dvPfee++999577733ujudTif33/8/XGZkAWz2zkrayZ4hgKrIHz9+fB8/Ih7/Hu8WZXqZ101RLT/7aHe881GaL6fVrFhefPbRuj3fPvjo9zj6jZPH2bQtLov2OqX2y+bRu2b22Ufztl09unv36upqfHVvXNUXd/d2dnbv/t5fPH89neeL7CPbuLi58XaxbNpsOc0/SpfZIv/so6f5qqyuF/myPamWbV2VZV4fKxYfpdS2XTefffR6PZ3mTfNRWhBCOx8Rqmn6uKwujvZ2du9t7+5s39tNd/ce7e4+2t9PP9nZebSzk569ePZlmqav2+wiT3cfpU+rq2VZZRhzusqmb+njJj2vq0X6Yv153qZNXoNCv3HSh/kgCnPvUfrVqgexrdI2qy8I4CKbzotlHoN4fycK8d6j9Az0KcuspZnqvbj36H4XFUe/dFotVmXe5r8rTeRdEIepZKdU6P2+5PiIRkM8MtPW28s1DW3bvLB5hm49R19kb4FGlpZF06bVeUBN0/VtJua4LN278+wyTyd5vrQg8plPmj5xDE0MjPT12+tmmi2XeT1+/RMnz1OSnPG9hzvj+3s+qc7zfPYofZNni5MbufY9qBKZoRsRElQ+VjFsCaVp0V4/Otjd3b2LD4/X7fxutlrd5Xm8e7l79xm98Zrmupjm4+ZyevfjGJ0tak9Pn3z1uUMtt6Rq0quCiD/JafRVTZ8Xy0fp549+31f5LPs8a/Pf96W0OyGhyG/TxbNiGYx8K2vbfLFq010wyP07MRifdmFU6+VtJvM2rGV5w4hZh5XuGl4Ssev8GZXC2yoQlcE1t/1mJLA7upd1vspq4EHdS0cWo9tQpz+O8Xh8G8WnbGH6jOqwPv2+CjDszSyNQWmXqiQ0zezyajGbX+WT3d298Sy/HDfupbKaZuWj3Z17ZKq+tvhakijznczzKau1q3nezvP6FnyYFg3JTUqtbeMp5CWtlin9QTr+vYfDc9DB9KCDaW/yNqFItP0GUXnYQeV1LlI/na+Xb0XUdw/e/8W9r/viva/74v7XffH+133x06/74oOv++LB4ItOrKMvPvy6L+7ufO03h3nnpjeHmeemN4e556Y3h9nnpjeH+ce9GX9zmIFuenOYg2568zYs1LEMG3RRIxr6fF2W12pByPP4EA31oSa948HfxjqndweM3Os2X8FT7xJA4ziPEOogrOoKfWw39OL2p/u72e7Ow/3tnYPJbHs/f3Bv+yDPPt2+v/Pppw92zu/t7+azr2vr7u925ur0XT5dt/D9COWb8b1NYMOjN+5Ax2/v0+o9ESAGyRCJWjeBmGJ372Zy3J4gl1m5JhYgzp+5CO2SXKxsUsZDwi6IJ/kF+QLu7Y3vGL+cW/cs+c+6Jd8cmDox7buEkTePGalq3a7WFFtRRqC6ah7dQtV0CeYFkj0hGm90/nugvcEIdsXyvKoXLONw2iY5IfnoNpDSbYp2hefAuY+Eq16/fvqT3/3i6be/e/qESH9LOGdN+un+9qRo0y9fK5w39ToWXG1+W9XGo/RZVja3fV1DxnRNeQIdyevf5/Wb0y9u+f7J81dGLhX3dB9TsnNv9+F4b+/+bcGs6xrzMSvqfEqB5/Wj9OTR7/tdih2JaX7f5pr0weLebQn65esuSukXxbSumuq8TRVm+uJN+ilxz4NPd3YtEWCq0t1b9iIMFHS0N6b/dsa7O7cBofZlg5B7zC1ZAWsaw6iiWJIKoBjdvf77Pj39Se9Phvb7bhaVrtry0Pt42HR/HA0Ou7BUDYt9LwQwjeLrw1USvqHBi2IBmlYrp1mdp9llVpT4MyrOXUg0n9+jFAcyHKfLy6KullASL0gevp9+ln5M5IzlUzaBUe1gQbBlel8gr/Iyz5r8xXoxyWsG45Hoa8Kq2rxhUO8L4KmX4l3gx9cDo/6gJUyHCb4muJ8UOfxAGnm4HS9nPsxh0XzvPurqp0nHufGD84W2N4EaAPU0b6Z1sWoNru8L5k3WvD2b8ast/dps39t/XxDKXLMn1wyGEpC/v0lavi8oIcWHgXKs6knz1xBDB+dl1jRXVS1U+pb3fA1YX5GtXZrpx/hm9qtbQ1P1/OFoKaAAJ1gP4PUTayiL57A59a0BElPO1lOh9uufeM4wOFv8nnDUKDs5+UVlF5Y2uTXIniJjG+6LubXgm2GqXaOlrksCQC+7tHm9XprcnzNNNEN5vdkQKUhC8xVBoDyuYDleNc4h2fze8WpF5nU8rZbnBeV/62zZwLm9VZBCr3+1mhFtUlpUeJ23iHiauwRqSfqFxve6RVKZs5rawVYTS93HAJ9Uq2vjsdxqpt6lT7M2m5A+sY6Ok5B0Va4RH2wtqxbIlsUUVp7T7XNy4fWN2yIHYr83pU+YAmvyMs7OXt/ynac5AmA7oPOi3EyMDj9UTXtLNJWKz6uKs9bEAvT2FQnLPCcOFUvRsKs/ew82U6gvKgthKk5cACKlfsD3tFxzK8fQRzJbXqfCWutaAjLLwvInSHYrZg6gOnamHrAwEWFpdG3k5uZelBSv4Q5wSixAGu8/ei9v/PelVMT+Mm8Vgw19/yz07X/VV7Dj/F3+c4KWAPh9Z/l5ti5/biijKFQU4dXFLG9+XyRS7F//r0Fp9/+NOO39vxGpT/9fiNT7EeqHg9P/6+hE+Y3/t6HEWdT/tyFVrShN1vy/jlaUl5z9v05LAan/1wkfkLq3AamfM6T2/9+I1P3/NyL1/zrlCaQe/L8RqYP/tyHVXOz+/v+v1FUGsb3/tyL2/zqd9f9GM3i2bPMLgfeG0Pt/ZTwRw/H/deIQRfL/dbY8iuUHWocfApb/L/Uhe3gCyf/3cyaw/H8/ZwLL/9epcE2KiQb6ucyQbcRs74eK2nuh9un/O1FDMH8zYns/fMSM5vt/JXLil/6/FrXbiMHPEWr3/t+L2v77ovbDQ+3+/3tRu41e+zlC7cH/e1E7+H8nai7m/n81ev8v0W9D6P2/VMe9rznV5ds3c7dMPs+adJLnS1qlbVpZcG6r98PvNsQ4O0+vq3V6Va3LWVoWb/O09ZBoK8IhwIBWjNOsbPN6SVS7zFMsB4B8o3RV5lmTp80qnxbn1wzmMquLbFLm6cev8tnnWZu/FLhPi5pWoqv6+mXWzj++DZq3WtPfvKDfJfYJDQkge+AepR+/D5XJ5US3v6/rvT+ivUf3P+1xIfr64K6IiRdE+Tafjd+j12rdrtbto9u88Rsnv3HyVYP5XuRp+DxKX79++pPf/eLpt797+mR3d+93+42T03f5dM3i9LIqi+m1bfnkepU1zW+cvKyrad40NI3H9XRetMQH6zqnBsdfPP10/zdOvlvVPNGWRQTA+zI+T2rDk/qT9AsQEjz2xjsYEf5TXKmzlWstLPBevfVnZTymyfhqqXKDHtJLxcJH8nVbrVb49vUvKn9iXbX5STad58+rbEYMmdeXxTSXRtv6Fyiuv308/E669fonng98d+fjtCGA6XlG2g4sc/yhY320t5NO51n9aI+k7hPLGwHav7BsD/3/p+nv1shXvz/Y6jdO8MYnJJEUntOUny3PK5/Lvlzly9O6rupHNLRrilUWY4Ws3GT+PKmWbV2Rpqof9T65k37Px+n7I0PKk2qxyJaz03fTfAXONdg8W5fl9U+ss7I4L/IZ9382wwwQFH31GVNx9EUxraumOm/HTpeMFSzh5trrZ9TDz9XkY1arFkodbLAipTlbs5qHuj4nQlVXwCkXap9Ia+aYYbCpzmVK7A1ttCbrkH48/vib5K6fRe46Kasm/38Pe53AEr+oWo9tvh6DnS2LtiDwDU0yuSIXNK+/cXJS5xmrvCmmkf4OXKBXebMuYRa+93o9xci/n76A2h+ee7/p01yUJyA5DYKXU3475ddTed+QxwegH93Y5RtKWBAJztPLe+Pd8e7Og/HOKB2/OH2TPqvp3SsyIenlPnTWzr3dh+O9vfui8OFerY1WTgv4V0QVuIJddX0zs//Gyat8UV2i8ellvmyfVxdpU63r6SbUqRsDZkMPjBgA1+iBZu+8rhYsoI1w5/h2cK7Ig2yEtudgtkF4fdrA+fQdi+OW2q2YcUhXmFZlBj/XmLbfOHnKRqHnWf/GyU+evnry5etTMvIBHPjNVzX5AOnH8sJd60mr0/xxelW08/Q3TnoN7saXPj72OvtSwTuEevAN+FtCv81IQq9/wwC0oXbVeW1gGGGrQew3gYZisHyerVbkpeks/TzSBa/WEgFmaVtnyyabEpsbtubhs0w8MeqBhUXJlq7mCHOqc/6w845H2tuoEK+5saEblMctxd4GjoHs2xhu7BmB22sueh/Mlx47hmF8RHGEpLGKI8BgRJHjjGkGI1W02rijhQVe0ILB9cDZtv0J7Gmv770++X56Ms+WF7mz25Cf11+dnJy+fn2bMOhVTnEK/JsZz30kaNy5DZjj9CqfGPFFxoG1NJys82q9nI1oJuiv9OzsdSc7cVVQV+S2rfKaYmCKdsfp70NxO8lVum4kaNcYm3yH4+/mk9fQRpQi+WkKo1wgXmi4ny1bDZcXxALpF9mSwvIaqqx5W6z6/W8OLjWifpoTxTmekjD//YK27ldj99V4uV697WdQIvS9VZ7gZdW0T20AcxuwLyoLYiqZkBAGzWoN/5mmkBnzMVTwkbh6afr4LvFncVm01/pR54PgT++P/ycAAP//ubBEevVOAAA=


When I decode that, I don't see any XML. All I get is a stream of bytes, mostly unprintable.

I get similar behavior when I use Raven's HTTP interface to fetch the same attachment.

This PowerShell script fetches the same attachment and prints the raw response:

Code: Select all
$Response = Invoke-WebRequest 'http://localhost:10300/static/attachments/tasks-34/output-log'
$Response.RawContent


The response looks like this:

Code: Select all
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Raven-Server-Build: 2666
ContentType: text/xml
Temp-Request-Time: 0
Date: Mon, 13 Jan 2014 18:43:44 GMT
Expires: Sat, 01 Jan 2000 00:00:00 GMT
ETag: 02000000-0000-0001-0000-00000000002B
Server: Microsoft-HTTPAPI/2.0

.?........??.`.I?%&/m?{.J?J??t?.?`.$??@.???????.iG#)?*??eVe]f.@??????{????{????;?N'????\\fd.l??J???!???.?~|.?\".?.?.ez??MQ-??hw??Q?/???X^|???=?>???8??????-.??:??????f??G??]
=?{???j|uo\\?.w?vvv???_<.=????#?????v?l?l9??J??\"????????.??=??m]?e^.+..???]7?}?z=??M?QZ.B;..?i???.??vv?m??l??Mw?.??>??O???y?????x?e?????\"Ow.?O??eYe.s???o??&=??E?b?y??M^?B?
q???
s?Q???.???6?/.?\"???e.?x.'
???? ?)?????????~..G?tZ-Ve???+M?].??d?T??????FC<2????5m???y?n=G_do?F??E???y@M??m&??,????2O'y?? ??O?>q M ?????f?-?y=~?.'?S?????;??{>???|?(}?g??.??=?.??...T>V1l   ?i?^?:?????
.????n?Z??y?{?{?.?????????rz??.?-jOO?|??C-??j??????i?UM?.?G???~?W?,?<k?????;!??o???b.?|+k?|?j?]0??;1.?vaT??m&?6?ey??Y???.^.???.???*.??5??f$?;??u??j?A?KG.??P????x|.??la???
?>??
0??,?Ai??$4???j1?_??????,?.7????f????{d????Z?(????)???y????.|?.?MJ?m?)?%??)?A:????s?????io?6?H??.Qy?A?u.R????oE?w.?????????????}???}????????????N??/>??/??|?7?y??7????7???
?7????7????..s??nzs??nz?6,?? .tQ#.?|]??jA???.??&?????:?w.???6_?S?.@?8?.? ??
}l7???????????????l{?.po? ?>??????.v???????k???v???]>]???.????M`??7?@?o???=. .?.?Z7??bw?fr?? ?Y?&. ???.??\\?lR?C?.?'?.?.????.??[?,???%?.?:1????7?.?j???.[QF??j.?B?t   ?.?=!.o
t?{???.v????.,?p?&9!??6??m?v???????^?~????????{??H.K8gM?????h?/_+?7?:.\\m~[????YV6?}]C?tMy..???????/n????WF..?t.S?so??xo??m???.?1+?|J?????????]?.?i~???????m   ???.J?.??????M
.f??M?)q??Owv-.`???[?\" .t?7??v??;?.??e??{?-Y.k.???X?
?.????>=?I?O???n.??????x?t...??T?}/.0????U.????b.?V+?Y???eV??3*?]H4???..2.??????P./H.??~?~L???S6?Q?`A?ez_ ??2????z1?k.???k????aP? ????]???.???%L?   ?&??.9?@.y?./g>?a?|?>???
I???????7?..?4o?u?j?? ?M??=???-??l??._.?\\?'? ?.???IZ?/(!???r??I??.C.?e?4WU-T???|X_??]????f??[CS???h)?.'X.??.k(???9??..S??S????x?08[??p?(;9?Ee.?6?5??\"c.??????a?]???K.@/??y
?^???3M4Cy??.)HB?.A?<?`9^5?!????jE?u<???.?.?l????U?B?.??.mRZTx???x??.jI?????ER?????V.K??.?T?k???j???O?6??>?????tU?..l-?.???.V???sr????\".b?7?O?.k?2??^????9.`;????L?.?TM{K4
????????.??.   ?<'..K????{.6S?/* a*N\\.\"?~???\\s+??G2[^??Z?Z.2???'Hv+f.?:v?.?0.aitm???^?.??.pJ,@.??z/o???T??2o.?}?,??.?W???]?s??.?}g?y?..n(?(T.???,o~_$R?_??Ai???8???.?O?_???.
???????D??????Y???!U?(M????V?????NK.???   .???.??3????????7\"??:?   ?.??.????!?\\??????U.????\"??:???F3x?l? ?????.e<.???u?.E??u?<??.Z?..??/?!{x.???s&??.?g.????pM??.????m?l????{?
???;QC0.3b{?|????.%r????.?????.j??????????C???{Q??^?9B???{Q;?.'j.??.5z?/?oC???T???9???7s?L>??t??KZ?mZYpn????6?8;O??uzU??YZ.o????h+?!??V???l?zIT??S,.?|?tU?Y???*?.?? ?2??lR
???????Y??.?O??V????e??????Z?????%?       {?.?.?.???D????????G???q!??????.D?6?????j????????.'?q?U??^?i?<J_?~????????{?dww?w????w?t????*???m??z?5?o????i?44???t^??.?:?.?_<?t?7N
?[?<??E.??2>Oj????? ..<??;..?S\\???k-,?^??ge<???j?r?.?K??G?u[?V???/*.b]??I6????lF ????4?F??.(??}<?N???'?.|w???!??yF?.,s??c}???N?Y?h???.?..???l.???????W????7N??'$?.????-?+?
??\\??????G4?k?U.c???d?<??m]???.?>??~????#C??j?????w?|.?5?<[???O???8/?.?.6? ..}?.Sq?E1???:o?N??.,????g?????cV?.J.l?\"?9[????>'BUW?).j?Hk??a???eJ?m?&??~<??????E?:)?&?..{??.?
?Z?m?.??-?? ?M2?\".4??qrR?.??)???.\\?Wy?.a.??z=???????.?{???\\?' 9??S~;??Sy???.?.???.JX.   ???{????????(.?8}?>???+2!??>t????????????^??VN ?WD.??]u}3?????|Q]???e?l?W.iS???&??..
fC.?..???f???.,??p??vp???l???`?Ax}????.??????qHW?Ve.?????8y?F??Y???O??z???S2?..??W5?.????]?I???qzU???7Nz???>>?:?R?;?z??[B??HB?.?.??v?ym`.a?A?7??b?|??V???,?<?.??..fi[g?&?.
?.????L<1???E????.s?s????G???.????.??-??.????.n?.??k.z.??.;?a|Dq????#?`D???i.#U?????.^????????   ?i???>?~z2??.??????_????~}?0?UNq
??.?}$h?????*?.?E???4???j???h&?????u';qUPW?????b`?v???Cq;?U?n$h?.?|???????F?\"?i
?\\ ^h??-[?.?.?.????.??y[???o..5?~?.?9??0?????Wc??x?^??gP\"??U??e??Om.s.?/* b*??..?j????.?1T???zi??.?gqY???Q???O???'..????Dz?N..


The content type says text/xml, but that sure doesn't look like XML!

Any ideas on how I should further decode this output?
Iain Elder, Skyscanner
isme
 
Posts: 83
Joined: Tue Jun 12, 2012 1:49 pm
Location: Edinburgh

Postby james.billings » Mon Jan 13, 2014 8:14 pm

Hmm, running the command against my DB gives a file that's all binary with no json at all. Curious.

I'll dig a little more at this end...
james.billings
 
Posts: 1120
Joined: Wed Jun 16, 2010 11:10 am
Location: Red Gate

Postby james.billings » Mon Jan 13, 2014 8:46 pm

OK, I needed to open the dump file inside dump.raven in 7Zip first. :)
james.billings
 
Posts: 1120
Joined: Wed Jun 16, 2010 11:10 am
Location: Red Gate

Postby james.billings » Mon Jan 13, 2014 9:14 pm

OK, some progress.
Like you, my unzipped file was full of this:


Code: Select all
{
      "Data": "H4sIAAAAAAAEAIVSy27bMBC8B8g/LHio2wMp0bKetWzUcRoUqFujbtAedGEoWhEqkYJIWfbfl45sN0WD5rRY7HA4s7PT+b6uYCdaXSqZIkpcBEJylZeySFFntjhC89n11ZRxU+5KcwCLlzrZ6zxFj8Y0ieP0fU96j6i2cMauS52fq88b/ihqhi7g8nUwLqU2THKBQLJapGipelkplkPX5MwIsNoIDQi1Ci3QdDpFm45zoTWC0qpxkdUJMK1UMRu71MMuxZ4L1Ev8MKET+PTl41cAONNag2fm0VI0lTrUQhpYMckK0Z5+G8G2VTWMnolvRY4L+4pwVTsDAVfStKrSTn7iJkw3+7kwPMV07IU09L3If8Pq5n3RWakPNKAR8wMcbPMYT6I4wCyPQ+xRL4wiP/b88YAe6FN/Ejy1vCqtxJSOrq9eMri8Xdzf/W2QhpR6bhTBw8EIDUbBTZL9KKUVqrPvt6t1Zupm8SEeE1tfYJ1c1rZhuyPh2aG4xDJwrltVtKxeMsOyb3Z4Z0fZn7Wetpo9zdeM/7LdDbO5Z2Sg0dm/GeBL4qTW5X/V3b8e5FsawGrxDvRwM9uuqg7P7BB7487xdI71fOyz31lQzmgfAwAA",
      "Metadata": {
        "ContentType": "text/xml",
        "Content-Type": "application/json; charset=utf-8"
      },
      "Key": "attachments/tasks-415/output-log"
    },



I took the data value, and Base64 decoded it here, saving the output to a .bin file

I renamed the bin file to .7z and opened it up in 7zip, then extracted out the contents. I now have this:

Code: Select all
<?xml version="1.0" encoding="utf-8"?>
<activity xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Download update 1.0.16.10" status="Success" id="0">
  <log>2013-01-30 13:57:14 INFO   Downloading update 'Deployment Manager 1.0.16' from 'http://www.red-gate.com/updatecontrols/download.aspx?etc=-1237175385&amp;guid=b1618a56-6fd9-4896-ad97-313788593525&amp;update=546&amp;client=1'
2013-01-30 13:57:14 DEBUG  Downloading 17113088 bytes to C:\\Windows\\TEMP\\tmpBA92.tmp
2013-01-30 13:57:44 INFO   Saving downloaded update to C:\\ProgramData\\Red Gate\\DeploymentManager\\Data\\PackageCache\\.updates\\Deployment Manager-1.0.16.10.msi
2013-01-30 13:57:44 INFO   Update 'Deployment Manager 1.0.16' (16 MB) successfully downloaded.
</log>
</activity>


Success!

So the steps seem to be:

- extract data
- unzip
- grab each data string
- base64 Decode
- unzip (again) the output

Hope that helps!
james.billings
 
Posts: 1120
Joined: Wed Jun 16, 2010 11:10 am
Location: Red Gate

Postby Mike Upton » Tue Jan 14, 2014 9:05 am

Amazing sleuthing without using the source code James :)

I was about to reply and tell you that, having looked at the source code for our log serialization, we GZip the XML string before it is saved as an attachment. RavenDB Smuggler then base64 encodes that to store it as a JSON text field in its dump file.

If you go straight to the HTTP API to download the attachment (http://hostname:10300/static/attachments/tasks-{foo}/output-log), you'll get back a binary stream that can be GZip decompressed to get the XML log.

You can also get a list of all the available attachments by reading the JSON array at http://hostname:10300/static
Mike Upton

Project Manager - SQL Compare|Data Compare|Comparison SDK
Redgate Software Ltd.
Mike Upton
 
Posts: 189
Joined: Wed May 11, 2011 8:04 am
Location: Redgate

Postby james.billings » Tue Jan 14, 2014 9:19 am

Actually I did poke the source code and saw some references to gzip so figured it was worth a punt... ;)
james.billings
 
Posts: 1120
Joined: Wed Jun 16, 2010 11:10 am
Location: Red Gate

Postby isme » Tue Jan 14, 2014 7:54 pm

Thanks for your help!

I distilled your helpful advice into a simple PowerShell function to export the deployment logs.

I ran it on the RGDM host to dump XML versions of the deployment logs to a log directory.

Code: Select all
PS C:\\Users\\admin_iain> Export-DeploymentLogs -TargetDirectory C:\\Users\\admin_iain\\Desktop\\Logs

PS C:\\Users\\admin_iain> Get-ChildItem -Path C:\\Users\\admin_iain\\Desktop\\Logs


    Directory: C:\\Users\\admin_iain\\Desktop\\Logs


Mode                LastWriteTime     Length Name                                                                                                                                         
----                -------------     ------ ----                                                                                                                                         
-a---        14/01/2014     18:43      61741 deployments-7966.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7967.xml                                                                                                                         
-a---        14/01/2014     18:43      57730 deployments-7968.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7969.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7970.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7971.xml                                                                                                                         
-a---        14/01/2014     18:43      50796 deployments-7972.xml                                                                                                                         
-a---        14/01/2014     18:43      62936 deployments-7973.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7974.xml                                                                                                                         
-a---        14/01/2014     18:43     137176 deployments-7975.xml                                                                                                                         
-a---        14/01/2014     18:43      50909 deployments-7976.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7977.xml                                                                                                                         
-a---        14/01/2014     18:43     136534 deployments-7978.xml                                                                                                                         
-a---        14/01/2014     18:43      51797 deployments-7979.xml                                                                                                                         
-a---        14/01/2014     18:43     136534 deployments-7980.xml                                                                                                                         
-a---        14/01/2014     18:43       6305 deployments-8001.xml                                                                                                                         



Now I can search the files to find what I need.

In case anyone else wants to do this, here's the code I used.

Code: Select all
function Export-DeploymentLogs {
  param (
    [String] $TargetDirectory = '.'
  )

  $AttachmentList = Invoke-RestMethod -Uri 'http://localhost:10300/static?start=0&pagesize=999999999'

  $AttachmentList |
  Where {
    $_.Key -like '*deployment*'
  } |
  Foreach {
    $AttachmentName = $_.Key.Split('/')[1]

    Invoke-WebRequest -Uri \"http://localhost:10300/static/$($_.Key)\" -OutFile \"$TargetDirectory\\$AttachmentName.xml.gz\"

    gzip.exe --decompress \"$TargetDirectory\\$AttachmentName.xml.gz\"
  }
}


It requires the gzip utlity to be on the path. I use version 1.2.4. You can download it from http://www.gzip.org/.

RavenDB pages the links to attachments through the static resource. Starting at 0 with the pagesize set larger than the number of attachments in the database, the static endpoint returns a link to all the attachments in the database.

The key property contains the word 'deployment' if the attachment is a deployment log. The function filters on this to ignore other types of attachment.

The full key is a relative URL for the attachment. If you fetch it, you get a gzip-compressed XML file.

gzip.exe deletes the input file and writes the decompressed version with an .xml extension.

In the end, the target directory contains a bunch of XML files.
Iain Elder, Skyscanner
isme
 
Posts: 83
Joined: Tue Jun 12, 2012 1:49 pm
Location: Edinburgh

Postby james.billings » Tue Jan 14, 2014 7:56 pm

Awesome, thanks for posting back your solution for everyone else to use if they need it!
james.billings
 
Posts: 1120
Joined: Wed Jun 16, 2010 11:10 am
Location: Red Gate

Postby isme » Thu Jan 16, 2014 2:14 pm

I added this script to a new Bitbucket repo called "RGDM Tools".

I'll copy the rest of my posted code there too.

Thanks again!
Iain Elder, Skyscanner
isme
 
Posts: 83
Joined: Tue Jun 12, 2012 1:49 pm
Location: Edinburgh


Return to Deployment Manager

Who is online

Users browsing this forum: No registered users and 0 guests