Steven's profileLunatic ExperimentsPhotosBlogLists Tools Help

Blog


    September 25

    The First Problem I Have Had With XboX Update

    I, like so many other people, have baught Halo 3 today.
    After inserting the game I decided that I may as well connect to Xbox Live. I failed to do so. The system says that there is a mandatory update available. I accept the update. Then, it says that the update failed. I retry about four times with out change.
    I decided to look up the problem. Maybe someone else has had this problem before and found the solution. I immediately checked support.microsoft.com for a solution. There is none. I then did a search on Google. Turns out that many people are having the same problem, and all entries I found are from today.
    I think some one messed up.

    One of the suggested remedies was to use the secret clear catch method. This can easily be found by googling the words xbox clear catch. Unfortunately it seems it didn't work for everyone.
    September 24

    How To Embed Post Compiled Code Into A Powershell Script

    My last post talked about how to embed a .NET assembly that has already been compiled into a POSH script. This time I will talk about how to embed code for a .NET assembly into a script that will be compiled when the script is executed.

    The .NET Framework comes with compilers for Visual Basic .NET and C# .NET. The .NET Framework also abstracts those compilers in the System.dll assembly through the Microsoft.VisualBasic.VBCodeProvider and Microsoft.CSharp.CSharpCodeProvider classes respectively.

    Using the CSharpCodeProvider class, I have created two scripts that make it easy to insert C# code into a POSH script. The Invoke-CScript script will compile it's code argument as a single method of a class and execute that method. The New-CAssembly script gives more direct access to the C# compiler allowing multiple, entire classes to be compiled. Once an assembly is compiled, it is loaded into the current application domain and reused whenever the scripts are invoked with an identical code parameter.

    Precompiled and post compiled assemblies have advantages and disadvantages.
    • A precompiled assembly is not dependent on availability of compilers, while a post compiled assembly must be written in a language that the end user has a compiler for. This shouldn't be a problem, however, if you program in Visual Basic or C#.
    • A precompiled assembly should load faster than the equivalent post compiled assembly since the precompiled assembly does not need to be processed by a compiler. This will only have a performance impact when the assembly is first loaded and will only be noticeable with very large assemblies.
    • A post compiled assembly can be inserted into a script as plain text, while a precompiled assembly must be maintained and compiled separately from the script.
    Do not over use embedded assemblies. Both methods of embedding an assembly into a script can result in a huge performance boost when hundreds of records must be processed. Both methods can also allow .NET classes to defined and used by the rest of the script. However, when no significant performance boost can be attained the result would only be an over complication due to having to use separate languages for compiled and interpreted code. The act of compiling the assembly will also take time.

    I have written and executed the following script as an illustration of the potential performance boost and as an example of how to use the two scripts. The script contains three functions that will use different methods to find the sum of the elements of it's input array.
    $CompileTime = Measure-Command {New-CAssembly @"
    namespace Test {
    public class SpeedTestClass {
    public static int GetSum(int[] arr) {
    int sum = 0; foreach (int i in arr) {sum += i;} return sum;
    }
    }
    }
    "@}

    "TestCModule compile time: " + $CompileTime.TotalMilliseconds

    function TestCModule([int[]]$in) {
    [test.speedtestclass]::GetSum($in)
    }

    function TestCMethod([int[]]$in) {
    Invoke-CScript 'int sum = 0; foreach (Object o in args) {sum += (int)o;} return sum;' $in
    }

    function TestPSMethod([int[]]$in) {
    [int]$sum = 0
    $in | Foreach {$sum += $_}
    $sum
    }
    [int[]]$arr = 1..10
    "TestCModule(1..10): $((measure-command {TestCModule($arr)}).TotalMilliseconds)ms"
    "TestCMethod(1..10): $((measure-command {TestCMethod($arr)}).TotalMilliseconds)ms"
    "TestPSMethod(1..10): $((measure-command {TestPSMethod($arr)}).TotalMilliseconds)ms"
    ''
    [int[]]$arr = 1..100
    "TestCModule(1..100): $((measure-command {TestCModule($arr)}).TotalMilliseconds)ms"
    "TestCMethod(1..100): $((measure-command {TestCMethod($arr)}).TotalMilliseconds)ms"
    "TestPSMethod(1..100): $((measure-command {TestPSMethod($arr)}).TotalMilliseconds)ms"
    ''
    [int[]]$arr = 1..1000
    "TestCModule(1..1000): $((measure-command {TestCModule($arr)}).TotalMilliseconds)ms"
    "TestCMethod(1..1000): $((measure-command {TestCMethod($arr)}).TotalMilliseconds)ms"
    "TestPSMethod(1..1000): $((measure-command {TestPSMethod($arr)}).TotalMilliseconds)ms"
    ''
    [int[]]$arr = 1..10000
    "TestCModule(1..10000): $((measure-command {TestCModule($arr)}).TotalMilliseconds)ms"
    "TestCMethod(1..10000): $((measure-command {TestCMethod($arr)}).TotalMilliseconds)ms"
    "TestPSMethod(1..10000): $((measure-command {TestPSMethod($arr)}).TotalMilliseconds)ms"

    The result:
    TestCModule compile time: 184.6108
    TestCModule(1..10): 4.0505ms
    TestCMethod(1..10): 146.3526ms
    TestPSMethod(1..10): 2.8179ms

    TestCModule(1..100): 1.1498ms
    TestCMethod(1..100): 25.9426ms
    TestPSMethod(1..100): 14.3305ms

    TestCModule(1..1000): 1.1065ms
    TestCMethod(1..1000): 25.8739ms
    TestPSMethod(1..1000): 187.4232ms

    TestCModule(1..10000): 4.1893ms
    TestCMethod(1..10000): 37.7209ms
    TestPSMethod(1..10000): 1589.6593ms

    The results of that test show that the code that made use of the class defined by the New-CAssembly compleated in just a couple of milliseconds. However, that is only after taking 184 milliseconds to compile that class. The code that used the Invoke-CScript returned in about 30 milliseconds in all but the first test. The large difference between using New-CAssembly and Invoke-CScript is that Invoke-CScript must make assurances that when the code is reused only the correct code is used, while with New-CAssembly we can simply create a unique type name and be fairly sure that when that type name is used we will be using the code that we want. The code that calculated the sum using Powershell syntax quickly began to take more time to process as the number of elements increased.

    Although it would appear that Invoke-CScript has a large overhead in comparison to New-CAssembly, in reality this overhead is due to having to make a call to New-CAssembly to verify the code that get executed. When you just want to execute fast code on a large number of records then using Invoke-CScript would probably be the best route, if you can suffice with a single invocation.
    # New-CAssembly.ps1
    # Version: 1.0
    # Author: LunaticExperimentalist
    # End User License: Public Domain

    param ([String]$Code = $(throw "C# code required to compile new assembly."))

    # get code hash
    $CodeHash = $Code.GetHashCode() -band 0x7fffffff

    # look for an existing assembly
    $ExistingAsm = [AppDomain]::CurrentDomain.GetAssemblies() | Foreach {
    $InfoType = $_.GetType("EmitedInfoClass")
    if (($InfoType) -and
    ($InfoType.GetProperty("ModuleCode").GetValue($null,$null) -eq $Code )) {
    return $_
    }
    }

    if ($ExistingAsm) {
    # use existing assembly
    return $ExistingAsm
    }

    # or compile a new assembly
    $CSharpCodeProvider = New-Object Microsoft.CSharp.CSharpCodeProvider
    $Parameters = New-Object System.CodeDom.Compiler.CompilerParameters(
    @([AppDomain]::CurrentDomain.GetAssemblies() | where {$_.GlobalAssemblyCache -eq $true} | Foreach {$_.Location}),
    $null,$false)
    $Parameters.GenerateInMemory = $true
    $CompileResult=$CSharpCodeProvider.CompileAssemblyFromDom($Parameters, @(
    New-Object System.CodeDom.CodeSnippetCompileUnit(
    'public class EmitedInfoClass { public static string ModuleCode{get{return "' +
    ($Code -replace '\\','\\' -replace '"','\"' -replace '\r','\r' -replace '\n','\n') + '";} } }'
    );
    New-Object System.CodeDom.CodeSnippetCompileUnit($Code)
    ) )

    # check for errors
    if ($CompileResult.Errors.Count -eq 0) {
    # use new assembly
    $CompileResult.CompiledAssembly
    }
    else {
    # write errors
    $CompileResult.Errors | Foreach {
    Write-Error "C Sharp Compile Error: Line $($_.Line) Col $($_.Column) `"$($_.ErrorText)`""
    }
    }
    # Invoke-CScript.ps1
    # Version: 1.0
    # Author: LunaticExperimentalist
    # End User License: Public Domain

    param ([String]$Code = $(throw "C# code required to compile new assembly."), [Object]$Arguments = $null)

    $Asm = New-CAssembly @"
    using System;
    namespace Emited {
    public class EmitedClass {
    public static Object EmitedCMethod(
    Object[] args,
    System.Management.Automation.PSObject[] input,
    System.Management.Automation.EngineIntrinsics context)
    { $Code }
    }
    }
    "@

    if ($Asm) {
    # use assembly
    $method = $Asm.GetType("Emited.EmitedClass").GetMethod("EmitedCMethod")
    $InputList = New-Object System.Collections.ArrayList
    $Input | Foreach {[Void]$InputList.Add($_)}
    $method.Invoke($null, @(@($Arguments),[System.Management.Automation.PSObject[]]$InputList.ToArray(),$ExecutionContext))
    }

    September 21

    How To Embed An Assembly Into A Powershell Script

    I may have gone too far this time in my far reaching experiments. The joy about script is that we can simply read them to find out whether they are safe to execute. The script that I am presenting today could make doing that difficult even for people that have advanced debugging techniques up their sleeves. The benefit, however, is that compiled code executes much faster than interpreted code, making this script useful in some cases.

    The script as you can see below is very simple. There are two parameters to this script the first is a list of paths that should point to valid assemblies, and the second is a switch that indicates that the loading process should not return the reference to the assembly once it finally gets loaded.
    The processes of the script is simple. For each path in the "AssemblyPath" parameter, the content of the file at that path is loaded as byte data. That byte data is then converted to a string using the Base64 method. that Base64 string is the prepended and appended with the code that will load the assembly. The result of that is then returned as a single string for each path that was given. The process of loading that assembly is also very simple. The Base64 string is converted back to byte data and sent to the "[AppDomain]::CurrentDomain.Load" method.

    The resulting string(s) are valid Powershell script and can be immediately exported to a .ps1 file or included in an existing script.
    This script will allow us to include precompiled, fast executing code into scripts, but beware, while this script is safe to use, it is very difficult to determine if a script that was made using this script by someone else is safe to execute. So unless you trust the life of your computer to the person that made the script, I would suggest inspecting any assemblies included in scripts you find before use.
    Something like the .NET Reflector should be of use if you can get the content of the embedded assembly to a file.

    # Get-AssemblyResource.ps1
    # Version: 1.0
    # Author: LunaticExperimentalist
    # End User License: Public Domain

    param ([String[]]$AssemblyPath, [Switch]$Silent)

    $AssemblyPath | Foreach {
    [String]::Join("`r`n", @(if ($Silent) {'[Void]('}
    "[AppDomain]::CurrentDomain.Load([Convert]::FromBase64String(@'"
    [Convert]::ToBase64String((Get-Content $_ -encoding byte),'InsertLineBreaks')
    "'@) )"
    if ($Silent) {');'}))
    }
    September 01

    Compressed Function Libraries

    The problem with Powershell scripts is that you can't run them while they are compressed. You may not be able to run them while they are compressed, but I can. I have written two scripts that can be used to import and export functions, and the exported functions can be compressed.
    The export function will read named functions out of the current runspace and save them as xml to a file. Those exported functions can optionally be compressed with GZip. The import function can then read the file and parse the xml to get to the functions. The imported functions can then be written back into the runspace or written to the output buffer.
    The GZip functionality uses the GZipStream that comes with the .NET Framework API so there no need to employ an external command for compression. GZip is also a very fast compression format and saving xml function libraries as GZip compressed will not cause significant overhead.
    There are two noteworthy limitations on the scripts. The first and more significant limitation is that the libraries are not yet checked for signatures. This means that if the import script can run, then any function stored in an xml function library can be run. The other limitation is that only GZip is supported. With modifications many different compression formats could be added, however, and I do intend to eventually add support for LZMA, the compression format that is used by 7Zip.
    Adding signature checking is my first priority for revising these scripts, and there is not a great amount of documentation on how that could be accomplished, so any comments on how to do so would be appreciated. Any general comments would also be appreciated. 
     
    # Export-FunctionLibrary.ps1
    # Version: 1.0
    # Author: LunaticExperimentalist
    # End User License: Public Domain
    
    param ([String]$Path, [String[]]$FunctionNames,
    	[String]$Encoding = 'utf8', [String]$CompressionFormat = 'None')
    
    if (($args -contains "-?") -or (-not $Path) -or (-not $FunctionNames)) {
    	"Export-FunctionLibrary"
    	"VERSION: 1.0"
    	"AUTHOR: LunaticExperimentalist"
    	""
    	"USAGE: Export-FunctionLibrary -Path [String] -FunctionNames [String[]]"
    	"       [-Encoding [String]] [-CompressionFormat [String]]"
    	""
    	"PARAMETERS:"
    	"  Path: A manditory parameter containing a path string to a file where the"
    	"        selected functions are to be stored."
    	""
    	"  FunctionNames: A manditory parameter containing a list of function names."
    	"                 The use of path style wildcards may be used here."
    	""
    	"  Encoding: An optional parameter selecting an encoding to be used when"
    	"            exporting the xml text file. Valid encodings are utf8, utf16le,"
    	"            and utf16be."
    	""
    	"  CompressionFormat: An optional parameter selecting a compression format to"
    	"                     be applyed to the resulting library. Only GZip is"
    	"                     supported by this version."
    	""
    	"DESCRIPTION: This script copys the code from one or more functions and saves"
    	"             them to an xml formated file. The resuling xml can optionally"
    	"             be compressed."
    	""
    	"SEE ALSO: Import-FunctionLibrary"
    	return
    }
    
    if (('none','gzip') -notcontains $CompressionFormat) {
    	throw 'Invalid compression format. Available formats are None and GZip.'
    }
    
    if (('utf8','utf16le','utf16be') -notcontains $Encoding) {
    	throw 'Invalid encoding. Available encodings are utf8, utf16le, and utf16be.'
    }
    
    function IsFunctionNamed {
    	param ([String]$FunctionName)
    	$FunctionNames | Foreach { if ($FunctionName -like $_) { return $True } }
    	return $False
    }
    
    function XMLEncode {
    	param ([String]$Text)
    	return $Text -replace '&','&' -replace '<','<' -replace '>','>'
    }
    
    # Generate the xml containg the names and script form the functions indicated by $FunctionNames
    $LibraryXML = [String]::Concat($(
    	''
    	Get-ChildItem Function: | Foreach { if (IsFunctionNamed($_.Name)) {
    		"$(XMLEncode($_.Name))$(XMLEncode($_.ScriptBlock.ToString()))"
    	} }
    	""
    ))
    
    # Encode the xml
    $Unicode = $( switch ($Encoding) {
    	utf8 {New-Object System.Text.UTF8Encoding($False); break}
    	utf16le {New-Object System.Text.UnicodeEncoding($False,$True); break}
    	utf16be {New-Object System.Text.UnicodeEncoding($True,$True); break}
    } )
    
    [Byte[]]$LibraryBytes = @($Unicode.GetPreamble(); $Unicode.GetBytes($LibraryXML))
    
    # Open a writable file at the path provided
    [IO.FileStream]$FileStream = $Null
    if (Test-Path $Path) {
    	# File exists
    	# Overwrive old file
    	$FileStream = (Get-Item -Path $Path).Create()
    }
    else {
    	# File does not exist
    	# Create new file
    	$FileStream = (New-Item -Path $Path -Type File).Create()
    }
    
    # Write the library byte data to the file
    if ($CompressionFormat -eq 'gzip') {
    	# GZip was requested
    	$GZip = New-Object System.IO.Compression.GZipStream($FileStream, 'Compress')
    	$GZip.Write($LibraryBytes, 0, $LibraryBytes.Length)
    	$GZip.Close()
    }
    else {$FileStream.Write($LibraryBytes, 0, $LibraryBytes.Length); $FileStream.Close()}
    
    # Import-FunctionLibrary.ps1
    # Version: 1.0
    # Author: LunaticExperimentalist
    # End User License: Public Domain
    
    param ([String]$Path, [String[]]$FunctionNames = '*',
    	[String]$CompressionFormat = 'None', [Switch]$List)
    
    if (($args -contains "-?") -or (-not $Path)) {
    	"Import-FunctionLibrary"
    	"VERSION: 1.0"
    	"AUTHOR: LunaticExperimentalist"
    	""
    	"USAGE: Import-FunctionLibrary -Path [String] -FunctionNames [String[]]"
    	"       [-CompressionFormat [String]]"
    	""
    	"PARAMETERS:"
    	"  Path: A manditory parameter containing a path string to a file where the"
    	"        selected functions are stored."
    	""
    	"  FunctionNames: An optional parameter containing a list of function names."
    	"                 The use of path style wildcards may be used here."
    	""
    	"  CompressionFormat: An optional parameter selecting a compression format to"
    	"                     be applyed to the resulting library. Only GZip is"
    	"                     supported by this version. The default value is none."
    	""
    	"  List: A swtch parameter that causes Import-FunctionLibrary to output the"
    	"        content of the library without creating new functions in the current"
    	"        runspace."
    	""
    	"DESCRIPTION: This script reads functions form a file and makes them available"
    	"             to the command line."
    	""
    	"SEE ALSO: Export-FunctionLibrary"
    	return
    }
    
    if (('none','gzip') -notcontains $CompressionFormat) {
    	throw 'Invalid compression format. Available formats are None and GZip.'
    }
    
    function IsFunctionNamed {
    	param ([String]$FunctionName)
    	$FunctionNames | Foreach { if ($FunctionName -like $_) { return $True } }
    	return $False
    }
    
    # Read text form file
    if (-not (Test-Path $Path)) {throw "Path not found."}
    
    $LibraryText=$Null
    if ($CompressionFormat -eq 'gzip') {
    	# Read GZiped content
    	$FileStream = (Get-Item $Path).OpenRead()
    	$GZipStream = New-Object System.IO.Compression.GZipStream($FileStream,'Decompress')
    	$FileReader = New-Object System.IO.StreamReader($GZipStream, $True)
    	$LibraryText = $FileReader.ReadToEnd()
    	$FileReader.Close()
    }
    else { $LibraryText = Get-Content $Path }
    
    # Parse text as xml
    $(
    	trap {throw "The library text could not be parsed as xml."}
    	$LibraryXML = [xml]$LibraryText
    )
    
    # Filter and apply functions from xml to the current runspace
    $LibraryXML.SelectNodes('/library/function') | Foreach {
    	if (IsFunctionNamed($_.Name)) {
    		if ($List) {
    			# Write functions to output
    			$Container = New-Object PSObject
    			Add-Member -MemberType NoteProperty -Name Name -Value ($_.Name) -InputObject $Container
    			Add-Member -MemberType NoteProperty -Name Script -Value ($_.ScriptBlock) -InputObject $Container
    			$Container
    		}
    		else {
    			# Make functions available for use
    			$FunctionName = $_.Name
    			$(
    				trap {Write-Error ('Could not import function "'+$FunctionName+'".'); continue}
    				Set-Content -LiteralPath ("Function:\"+$FunctionName) -Value $_.ScriptBlock
    			)
    		}
    	}
    }