Import Datasource via API and PowerShell

Locked

Discussion interactions are currently paused.

Hello All,

I'm attempting to import datasources into our LogicMonitor instance, using the API and PowerShell. The following documentation only provides a CURL example, which isn't really sufficient for us.

https://www.logicmonitor.com/support/rest-api-developers-guide/datasources/import-datasources-from-xml/

Usage:

Import-LMDatasource -Credential $credentials -FilePath 'c:\repositories\LogicModules\DataSources\CustomDataSource.xml'

I am assuming that you have an array containing your accessId, accessKey and company name, prior to calling the function. The parameter FilePath is the full path to the XML file to be uploaded.


<span class="pln">function Import-LMDatasource
{
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $false)]
        [array]$Credential,
        [Parameter(Mandatory = $false)]
        [ValidateNotNullOrEmpty()]
        [string]$FilePath
    )
    Begin {
        if(-not($Credential))
        {
            $accessId = Read-Host -Prompt "Please supply accessId:"
            $accessKey = Read-Host -Prompt "Please supply accessKey:"
            $company = Read-Host -Prompt "Please supply company:"
        }
        else {
            $accessId = $Credential.accessId
            $accessKey = $Credential.accessKey
            $company = $Credential.company
        }

        $httpVerb = 'POST'

        $resourcePath = '/setting/datasources'        
        $queryParams = '/importxml'
        $data = ''

        $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams
        $epoch = [Math]::Round((New-TimeSpan -Start (Get-Date -Date '1/1/1970') -End (Get-Date).ToUniversalTime()).TotalMilliseconds)
        
        $contentType = [System.Web.MimeMapping]::GetMimeMapping($FilePath)
    }
    Process 
    {
        $requestVars = $httpVerb + $epoch + $data + $resourcePath
        $hmac = New-Object System.Security.Cryptography.HMACSHA256
        $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey)
        $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars))
        $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-'
        $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower()))
        $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
  
        Add-Type -AssemblyName System.Net.Http

        $httpClientHandler = New-Object System.Net.Http.HttpClientHandler
        $httpClient = New-Object System.Net.Http.HttpClient $httpClientHandler
        $httpClient.DefaultRequestHeaders.Authorization = $auth
        $packageFileStream = New-Object System.IO.FileStream @($filePath, [System.IO.FileMode]::Open)

        $contentDispositionHeaderValue = New-Object System.Net.Http.Headers.ContentDispositionHeaderValue 'form-data'
        $contentDispositionHeaderValue.Name = 'file'
        $contentDispositionHeaderValue.FileName = (Split-Path -Path $FilePath -Leaf)

        $streamContent = New-Object System.Net.Http.StreamContent $packageFileStream
        $streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue
        $streamContent.Headers.ContentType = New-Object System.Net.Http.Headers.MediaTypeHeaderValue $contentType

        $content = New-Object System.Net.Http.MultipartFormDataContent
        $content.Add($streamContent)

        $response = $httpClient.PostAsync($url, $content).Result

        if(!$response.IsSuccessStatusCode)
        {
            $responseBody = $response.Content.ReadAsStringAsync().Reult
            $errorMessage = "Status code {0}.  Reason {1}.  Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody
            throw [System.Net.Http.HttpRequestException] $errorMessage
        }

        return $response.Content.ReadAsStringAsync().Result

        # $httpClient.Dispose()
        # $response.Dispose()
    }
    End 
    {

    }
}</span>

Result:

{"errmsg":"Request content is too large, max allowed size is 10240","status":1007}

Dot Sourcing the function at runtime allows me to inspect the variables set during execution:

$response


<span class="pln">Version             : 1.1
Content             : System.Net.Http.StreamContent                                                
StatusCode          : OK
ReasonPhrase        : OK                         
Headers             : {[Date, System.String[]], [Server, System.String[]]}                         
RequestMessage      : Method: POST, RequestUri: 'https://</span><span class="tag"><instance></span><span class="pln">.logicmonitor.com/santaba/rest/setting/datasources/importxml', Version: 1.1, Content:          
                      System.Net.Http.MultipartFormDataContent, Headers:
                      {
                        Authorization: LMv1 </span><span class="tag"><ACCESSTOKEN-OBFUSCATED></span><span class="pln"> 
                        Content-Type: multipart/form-data; boundary="17ba48f6-b5e9-48c6-9002-7544d99025d8"
                        Content-Length: 11858
                      }   
IsSuccessStatusCode : True</span>

Uploading the exact same datasource XML via the GUI works. I think I'm most of the way there, but obviously something I'm doing is bloating the request size and tripping this limit.

Has anyone had any success with uploading datasources via the API?

Many Thanks,

~Nick

nrichards

Posted 9 years ago·Last reply 7 years ago

4 comments

Preston_Gallwas

·7 years ago

Well, the Datasources "import" from file option pointed me in the right direction. Turns out my XML file wasn't valid (it had a wrong encoding). I fixed that and will give this another go the next time I have a datasource to import.

nrichards

OP9 years ago

Hi Sarah,

Thanks for this. My import is now working following the example you provided. Though I am getting some formatting issues for the Collector Attributes > Groovy Script section. I don't know if its actually an issue yet, but imagine that it might be.

I understand your comment completely;

Quote

"This one is a bit tricky - we didn't publish much around this resource because there wasn't a lot of interest and also it starts to overlap with where the LogicModule Exchange is useful.

We're looking to commit our internally amended / created data sources into version control, and import them into our production instance under change, hence the desire for this.

Many Thanks,

~Nick

Sarah_Terry

·9 years ago

Hi Nick,

This one is a bit tricky - we didn't publish much around this resource because there wasn't a lot of interest and also it starts to overlap with where the LogicModule Exchange is useful. That being said, we'll update the docs to provide more useful examples. We previously only supported Basic Auth for this resource, but have since added support for authenticating via API Tokens. In order to get the token authentication to work, you need the following:

(1) the payload needs to include the xml file content enclosed in boundaries (can be random)
(2) the content type needs to be set to multipart/form-data and it needs to call out the boundaries

And in your code above, you'll want the /importxml to be tacked on to the resource path (I recognize that doesn't align with the proper REST resource naming, and we'll improve it in a future release). So a successful example in PowerShell looks like this:


<span class="pln">  
<# account info #>
$accessId = '48v2wRzfK94y53sq5EuF'
$accessKey = 'aB=k9opY87U~^_cvHiq~98G-j6Q_ja4sr6J8-4+)'
$company = 'api'

<#File & Request info#>
$uploadFile = 'test.xml'
$FilePath = "C:\Users\Sarah\Desktop\test.xml"
$httpVerb = 'POST'
$resourcePath = '/setting/datasources/importxml'        
$queryParams = ''

$file = Get-Content $FilePath

$boundary = [System.Guid]::NewGuid().ToString()

$data = '------$boundary
Content-Disposition: form-data; name="file"; filename="test.xml"
Content-Type: text/xml

'+$file+'

------$boundary--'

<# Construct URL #>
$url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams

<# Get current time in milliseconds #>
$epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds)

<# Concatenate Request Details #>
$requestVars = $httpVerb + $epoch + $data + $resourcePath

<# Construct Signature #>
$hmac = New-Object System.Security.Cryptography.HMACSHA256
$hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey)
$signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars))
$signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-'
$signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower()))

<# Construct Headers #>
$auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization",$auth)
$headers.Add("Content-Type",'multipart/form-data; boundary=----$boundary')

<# Make Request #>
$response = Invoke-RestMethod -Uri $url -Method $httpVerb -Body $data -Header $headers 

<# Print status and body of response #>
$status = $response.status
$body = $response.data

Write-Host "Status:$status"
Write-Host "Response:$body"

</span>

Let me know if this doesn't work for you, or if you have additional issues/feedback.

Thanks!

Sarah